Simplify Testing using Fault Injection

Presentation Slides

Example Source

Below are instructions on using the examples in Eclipse and AspectJ. (Please note that the examples are specific to AspectJ but neither AspectJ nor the examples are specific to Eclipse.)

  • Download, install and run Eclipse 3.2.1
  • Download and install AJDT (which includes AspectJ) through the download manager (Help -> Software Updates -> Find and Install -> Search for new updates to install. Add a “New Remote Site” with the URL http://download.eclipse.org/tools/ajdt/32/dev/update). Please note that the examples require the lastest development build due to a bug in AJDT.
  • Download and unzip the examples into a convenient location (you can use the workspace directory for example).
  • Within Eclipse, import (right click on the Package Explorer and select Import -> General -> Existing Projects into Workspace) the three example projects (“Example”, “fault.injection” and “javassist”). (If you select the directory where you unzipped the files then the three projects should appear in the dialog.) (It may take a moment or two to set up and build everything. There should be no errors (though there will be warnings).)
  • You can run the example with out any faults (right click on “Main.java” in the “Example” project in the Project Explorer and select “Run as” -> “Java Application”).
  • To run the examples with different faults requires a bit of setup:
    • Select “Run…” from the toolbar (it’s a drop down from the green play-like button).
    • In the dialog click on “AspectJ / Java Application” in the left navigation and right click “New”.
    • Under “Project” set it to fault.injection and under “Main class” set it to com.example.Main
    • Switch to the “Arguments” tab (still in the Run dialog) and add to the “VM arguments”: “-javaagent:lib/aspectjweaver.jar” (without the quotes). (This is the JVMTI agent that will load-time weave the aspects.)
    • Click “Run”. At this point the console should show the same information that you saw when you ran “Main.java” directly. This is because the META-INF/aop.xml doesn’t have any aspects configured.
    • Open the META-INF/aop.xml which is located in the “fault.injection” project under “classes”. (Because Eclipse tries to mask some complexity from you it’s difficult to get at the “classes” directory in the IDE unless you set up some filters correctly. For simplicity, just open the file in some text editor via the filesystem.)
    • Uncomment the example fault that you are interested in (i.e. remove one set of “!–” and “–” from the aspects section of the file) and save the file. When you re-run the application in Eclipse the load-time weaver will weave in the aspect that you uncommented. (If something doesn’t work as expected then check the top of the console output. The weaver may have some information for you.) Please note that in one case there are two aspects (“Read2” and “Read2Helper”) as part of one example.

As I mentioned in the presentation, to apply techniques similar to this in your own environment, I would recommend using Ant to generate an appropriate META-INF/aop.xml and then run JUnit in a loop to run through different faults.

I tried to structure the projects to decouple them as much as possible. For example, the “fault.injection” project doesn’t directly depend on the “Example” project. (It only depends on the output classes.) I could have gone a step further and generated a JAR out of the “Example” project and used that in the “fault.injection” one. The goal to all of this was to ensure / prove that load-time weaving was being done.

Please don’t hesitate to contact me if you have questions or problems.

Advertisements

9 comments

  1. I read most of your Fault Injection presentation. Very engaging to say the least from a technology / implementation-independent perspective.
    Here are a few thoughts on your Java examples, given that I stopped using Java quite some time ago in favor of Ruby.
    In Ruby, I don’t need to learn a special frameworks or ready any O’Reilly books to start. Since Ruby classes are open, I can simply re-define *any* method, even those that are at the system interface/API level. If I want File.delete() to puke chunks, I simply redefine it to raise Chunks. done. This makes fault injection testing almost trivial.
    Ok, I don’t want to start (another) Java vs. Ruby holy war. But I really think your presentation is good, but [c|w]ould be more easily implemented in a language like Ruby.
    word.

  2. Hey hey! Thanks for the comments Dave.
    I just want to point out one concern about your comment: if I have a trunk load of code in Java that I need some snazzy technique (such as fault injection) to effectively test, then I’m unfortunately stuck using Java with all of its warts. If I can choose the language that I want to use for coding in a particular domain then I might choose Ruby since it would be much easier to test using techniques like fault injection. I want to be clear that combining the two at this time is not an option (i.e. code in Java and somehow test via Ruby).
    I also have one question: The focus of my presentation was that you can use the technique in a completely non-invasive way — that is, no code (in most cases) needs to be touched. Is this possible with Ruby? Can I take already written Ruby code and, at load-time, decide which methods I’m going to redefine and with what? If it is possible then I would think that a nice write-up is in order. We should talk.

  3. Rob,
    I can very definitely say it can be done very easily in Smalltalk (decide at load time what methods to redefine and with what)
    Regards
    Anand

  4. Thanks for the info Anand. Given the Smalltalk’y flare to Ruby, I’m betting that it’s also possible.
    Method overloading / overriding is just one of many needed techniques. Can you provide some info about updating the state of other objects specifically by intercepting and overriding member assignments? Overloading object creation? Thanks!
    There were two goals to my presentation (and an article hopefully soon): the first and primary goal was to get people thinking about how to test various aspects of their code that are traditionally quite difficult and the second was to provide various techniques to do this in Java. I’m glad to find out that the concept is sound in other languages.

  5. Rob,
    SMalltalk has very powerful reflective capabilities. As an example, I can programmtically examine/remove/change method definitions of every class in the system, let alone member assignments. overloading object creation would be trivial to do
    As a small example lets say we have a class Foo (subclass of a abstract base class Object) with the methods zot and new defined on it –
    The definition would be along the lines of
    Foo subclass:#Object
    instanceVars:#(zot)
    ….
    Foo >>zot
    ^ zot “getter”
    Foo>>zot: aZot
    zot := aZot “setter”
    Foo class >>new “equivalent of a static method in Java”
    ^ super basicNew
    zot: nil;
    yourself
    Now having this defined in my system, I can do
    |aDict aCompiledMethod|
    aDict := Foo methodDictionary “retreive the list of methods defined for Foo”
    aCompiledMethod := aDict at: #zot.
    “aCompiledMethod is an instance of a class that holds a) the byteCode b) the source (text) of the method definition c)list of methods which are called from the method and so on ..
    I can supply a new source as a string and compile it on the fly and magically the runtime behavior changes.
    Another interesting thing i can do is to remove the method and then do something like
    [Foo zot]
    on:DoesNotUnderStand
    do:[:ex| … “redefine zot as I see fit”]
    If I wanted to get still more fancy, I could, at load time, change all instances of CompiledMethod on the classes that interest me to an instance of my own class – say CompiledMethodWrapper. I could then using some common sense approaches add some extra behavior at that level to each method …
    Dont know if I have explained myself clearly. This can feel like black magic to people who havent worked in Smalltalk. It is quite simple actually
    Feel free to drop me a line if u need more details
    Also, you may want to check this link which talks about interception – look at the section on pre an post conditions …
    http://www.ifs.uni-linz.ac.at/~ecoop/cd/papers/1445/14450396.pdf
    Hope this helps
    Regards
    Anand

  6. Thanks again Anand and yes, you’ve made the approach pretty clear. It’s been too long since I’ve used Smalltalk and it’s been too long that I’ve been working in strongly typed languages.

  7. Rob,
    I asked one of colleagues if Ruby supported what you asked for – he replied it was very doable. As an example, here is some code he provided to intercept a method call – adding methods/classes is similarly easy
    ————–
    class A
    #This method will intecept calls to methods present in methodsToIntercept Array,
    #it will first call methodToCall and then call the intended method.
    def A.intercept(classVar,methodsToIntercept,methodToCall)
    methodsToIntercept.each do |method|
    #Stores the old method inside a variable
    old_method = instance_method(method)
    #It defines a new method by same name as the method to be called and passes a block which is the definition of the new method.
    define_method(method) { |*args|
    #Calling methodToCall
    send(methodToCall,”Method “+method.to_s+” called”)
    #Calling old method
    old_method.bind(self).call(*args)
    }
    end
    end
    def method1(*args)
    puts “In method1”
    end
    def method2(*args)
    puts “In method2”
    end
    def log(message)
    puts “Logging Message..”
    puts message
    end
    end
    #Calling intercept method with arguments defining which methods to be intercepted and which method to be executed before calling the intended method.
    str = %q{intercept(self,[:method1,:method2],:log)}
    A.class_eval(str)
    #Instantiating Class A
    aObj = A.new
    #calling method1
    aObj.method1
    ———
    Regards
    Anand

  8. Thank you yet again Anand! When I get a few minutes I’ll post that in an article so that it gets good formatting. If there’s any code (Smalltalk, Ruby or otherwise) that you want to email me, I’ll get it posted so that there are code samples for languages other than Java.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s