InsectJ Manual 9-28-2005 (version 1.0.6) InsectJ is a tool to instrument programs using byte code rewriting, mainly to record information while running the application, such as coverage information. InsectJ is designed to be very easy to extend with different kinds of monitoring and different types of instrumentation. The type of instrumentation is lousely coupled with the monitoring of that instrumentation. The user is encouraged to implement its own monitoring on a case by case basis. One monitor can monitor multiple types of instrumenation, by implementing the monitor methods for each monitor interface the different instrumentation types declare. This manual will explain how to instrument a system using the Eclipse plugin or without. How to create new monitors and new types of instrumentation, called probe inserters. ------------------------------------------------------------------ Instrument a program ------------------------------------------------------------------ The instrumentation of a program takes four entities. - The program which needs to be instrumented. - The instrumentor (InsectJ). - A type of instrumentation (probe inserter). - A monitor which processes the information created by the instrumentation. The easiest way to instrument a program is by using the Eclipse plugin. The program which you want to instrument is normaly inside a project as source code, byte code or in a jar, it doesn't matter. The instrumentor has to be on the classpath of the program. This can be done by selecting "Add InsectJ to classpath" when right clicking on the project in the Package Explorer. This will add the instrumentor and all bundled probe inserters and monitors to the projects classpath. To instrument and run the program, go to the launch wizard and select "InsectJ Application" and "InsectJ run" and create a new Configuration. The first tab is identical to a normal Java application run. Select the project and entry class. The second tab contains the instrumentation configuration. The instrumentation configuration is stored in a XML file, by default it is called "insectconfig.xml" and stored in the project root. Later we can use it to instrument projects without Eclipse. For every type of instrumentation we want to apply to our program we need to add a probe inserter to the configuration and configure it. After selecting add (or selecting edit when on a already added probe inserter) you need to select one and configure it. These are the settings: - probe inserter: The type of instrumentation you want to add. - monitor: Almost every probe inserter needs a monitor. Select one from the dropdown box. Only monitors which are suitable for the probe inserter can be selected. - enabled: Used to enable and disable an instrumentation without removing it. - factory (optional): If you want to use one monitor type for multiple types of instrumentation, or change the creation of the monitor objects somehow, you can select a monitor factory here. You can create a factory by implementing MonitorObjectFactory in the framework. - arguments (optional): Some probe inserters can take arguments as a string to customize the instrumentation. For example, the method entry instrumentor can be set to collect nothing (none), the arguments (args), the instance (this) or everyting (all). After selecting the type of instrumentation and the monitors. You have to select which elements to instrument. Open the project you want to instrument, and select the individual source files and packages. You can also select the individual methods and constructors. Please note, this feature is still experimental. Sometimes not everything below a node in the tree is selected when the node is. Also, large projects or packages can make the system very slow, because it keeps long lists of classes in memory and updates those every time. This is true for instrumenting the API. At the moment the instrumentor does not instrument the API, even if it is selected here. Last there are some settings for the instrumentor. - Verbose: When selected the instrumentor will echo certain operations on the console. - Store instrumented files to folder: Every instrumented class will be stored in the specified folder in the current project. Useful for debugging new instrumentation and running the instrumented program outside the instrumentor. - Save configuration file: Selects the name to store the configuration. This can be used to later load the configuration file again, or use it for instrumentation outside Eclipse. ------------------------------------------------------------------ Simple monitor ------------------------------------------------------------------ A simple monitor is an implementation of one monitor interface for the monitoring of only one probe inserter. In Eclipse the user can use the "new"-wizard, and create a new "Concrete Monitor". The user needs to specify a name and a location for the new monitor, and what type of instrumentation it wants to monitor, by selecting the interface type it wants to implement. For example, we make a MyCastMonitor which implement the CastMonitorInterface, this is the stub class created: public final class MyCastMonitor extends AbstractMonitorObject implements MonitorObject, CastMonitorInterface { public MyCastMonitor(Class parent, int probeCount, Class monitorInterface) { super(parent, probeCount, monitorInterface); // TODO Auto-generated constructor stub } @Override public void processData() { // TODO Auto-generated method stub } public void reportCheckCast(Object obj, Class cast, int probeId) { // TODO Auto-generated method stub } public void reportCast(Number after, Number before, int probeId) { // TODO Auto-generated method stub } } The two methods called reportCast and reportCheckCast will be called during the execution of the program instrumented with this monitor and the cast probe inserter. The processData method will be called when the program ends, this can be used to report the collected data in some way. The constructor will be called by the classes which are instrumented with this monitor. It will be called after they are loaded and it gives some information about the instrumentation. The "parent" argument contains the class file which this instance of the monitor is responsible for. When the AbstractMonitorObject class is used as the super class of your monitor, it is not needed to store this in the concrete monitor, because it will be available as the public final field called "monitoredClass". The "probeCount" argument tells how many different probes there are in this class. This can be used for coverage information. Again, this value is available as a public field inherited from AbstractMonitorObject with the name "probeCount". Finally, the "monitorInterface" class is the interface which is used by the probe inserter in this instrumentation. It will be clear in the more advanced monitor interface why this is needed. This value is not stored in the AbstractMonitorObject. The report methods have arguments which are related to the instrumentation. For example, the reportCheckCast has two arguments which relate to the cast operation. When an object is cast to some class, this method will be called, and the "obj" argument will hold the object being cast, and the "cast" argument will contain the class object which the object is being cast too. The "probeId" argument gives the ID value for the current probe in this class. This ID can be used to get information about this particular probe and the method which holds it. There are currently two methods in the AbstractMonitorObject class which use this ID value, "getMethodString(int)" and "getContext(int)". The former method returns the declaration of the method which holds this probe as a java method declaration. This can later be parsed to extract specific information. We plan to add this parsing to the framework itself in the future. "getContext(int)" returns information particular to the probe which can be different for every probe inserter. For example, the basic block probe inserter stores the source line numbers which correspond to the basic block which is represented by this probeId. The cast probe inserter doesn't store any extra context information. As an example, I give an implementation of a cast monitor which checks if precision is lost by a cast and prints all the cases where some precision is lost at the end of the program. private StringBuilder builder = new StringBuilder(); @Override public void processData() { System.out.println(builder); } public void reportCast(Number after, Number before, int probeId) { if( before.longValue() != after.longValue() || before.doubleValue() != after.doubleValue()){ StringBuilder b = builder; b.append("Precision loss in : "); b.append(monitoredClass.toString()); b.append(" "); b.append(getMethodString(probeId)); b.append('\n'); b.append("Before cast: "); b.append(before); b.append(" after: "); b.append(after); b.append('\n'); } } ------------------------------------------------------------------ Complex Monitor ------------------------------------------------------------------ One monitor can implement multiple monitor interfaces. Because the constructor is fixed by the AbstractMonitorObject, the system can not pass the information for all probe inserters to the monitor at once. When a monitor class implements multiple monitor interfaces and the program is instrumented with the corresponding probe inserters, one seperate instance is created for every class / probe inserter combination. This situation is undesirable if the operation of the monitor depends on the interaction between multiple probe inserters. That is the reason why there are two additional ways to create monitor instances during the execution of the program. One uses a static method in the monitor class and the second uses a seperate factory class. Here is an example of a monitor which uses the method entry and exit probe inserter to start and save the recording of which branches are taken by each invocation of the methods. public final class BranchEnumerator extends AbstractMonitorObject implements BranchMonitorInterface, MonitorObject, MethodExitMonitorInterface, MethodEntryMonitorInterface { private int index; private BitSet currentTrace; /** * For every method a list of branch traces is kept * The index is the method exit probe id */ private Map> branchTraces = new HashMap>(); private static HashMap cache = new HashMap(500); public BranchEnumerator(Class parent, int probeCount, Class monitorInterface) { super(parent, probeCount, monitorInterface); } public static BranchEnumerator create(Class parent, int probeCount, Class monitorInterface){ BranchEnumerator mon = cache.get(parent); if(mon == null){ mon = new BranchEnumerator(parent, probeCount, monitorInterface); cache.put(parent, mon); } return mon; } @Override public void processData() { System.out.println("Branch Enum for " + getClassName()); for (Map.Entry> entry : branchTraces.entrySet()) { System.out.println(getMethodString(entry.getKey())); for (BitSet trace : entry.getValue()) { System.out.print('\t'); System.out.println(trace.toString()); } } } public void reportBranch(float distance, int probeId) { currentTrace.set(index++, distance > 0); } public void reportBranch(int distance, int probeId) { currentTrace.set(index++, distance > 0); } public void reportBranch(boolean taken, int probeId) { currentTrace.set(index++, taken); } public void reportSwitch(int target, int probeId) { currentTrace.set(index++, target > 0); } public void reportExit(int probeId, Object return) { reportExit(probeId); //unused } public void reportExit(int probeId, Throwable ex) { reportExit(probeId); //unused } public void reportExit(int probeId) { List traces = branchTraces.get(probeId); if(traces == null){ traces = new LinkedList(); branchTraces.put(probeId, traces); } traces.add(currentTrace); } public void reportEntry(int probeId, Object instance, Object[] args) { reportEntry(probeId); //unused } public void reportEntry(int probeId, Object instance) { reportEntry(probeId); //unused } public void reportEntry(int probeId, Object[] instance) { reportEntry(probeId); //unused } public void reportEntry(int probeId) { index = 0; currentTrace = new BitSet(); } } I'll go over the various parts of this monitor class to explain what they do. The idea is that when a method is entered, a BitSet is created which stores the direction of the branch which is taken. The private variables "index" and "currentTrace" store the index in the bitset and the bitset itself. In "reportEntry()" it can be seen we initialize these variables. "reportExit()" contains the code which stores the created bitset in a Map. There is one Map per monitor, and thus per Class, so we have to use the method as index for the map, and a list for all the individual traces. We could have used the method name as index, but I decided to use the probeId value from the exit method probes, which are unique for every method. The "reportBranch()" (and "reportSwitch()") methods are called at every branch. There is a different one for every type of branch, but we do almost the same thing for every branch. We store at the index which direction we took and update the index. The methods for entry and exit of methods also have different versions, which one is used can be selected by setting the arguments for the probe inserter, but we do not need the extra information, so we'll use "none" and no extra information is gathered and the simplest version is called. Because normaly, a monitor instance is created for every probe inserter class combination, there would be 3 instances of the monitor for every class, and each would get there respective calls. This is fine if they communicate using some static data structure which is global for all instrumented classes but not in our case. That is the reason for the "create(Class, int, Class)" method. This is a factory method, which creates monitor and caches them in a Map. It makes sure that there is only one monitor instance of every class. The third Class argument contains the Class object from the interface the current probe inserter uses. This can be used to discriminate between the different probe inserters when they tell the monitor how many probes the probe inserter inserted in the class. Finaly there is the "processData()" method, which all monitors have. This method is called when the application quits. We use it here to output the collected information to System.out, but alternatively, many other things can be done, like storing it in a database or file, anything is possible. ------------------------------------------------------------------- Custom Probe Inserters ------------------------------------------------------------------- The InsectJ system is designed to be extended with new probe inserters. All the buildin probe inserters use the same mechanism as any custom probe inserters. In contract with the monitors, probe inserters use the plugin system from Eclipse, so to create a new probe inserter, the developer of the probe inserter needs to create a new plugin project. The project has a dependency on the edu.gatech.cc.rtinsect plugin, and it has to add the "bcel-5.1.jar" which is inside the rtinsect project to the classpath. To create a new probe inserter, two class files need to be created. The probe inserter class, which needs to extend from "AbstractProbeInserter" and the monitor interface, which needs to extend "MonitorObject". An extension has to be made at the "edu.gatech.cc.rtinsect.probeInserter" extension point from the "edu.gatech.cc.rtinsect" plugin. Make a new probe by right clicking on the extension. These are the values which are needed: * name - A human readable name for the probe inserter * id - The unique identifier for the probe inserter * class - The fully qualified name of the class that extends edu.gatech.cc.rtinsect.AbstractProbeInserter. * monitorInterface - The fully qualified name of the interface which this probe inserter declares to be implemented by concrete monitor. Needs to extend edu.gatech.cc.rtinsect.MonitorObject. * description - The description of what this probe inserter inserts. * jarLibrary - The path to the jar file which contains the runtime classes of the probe, relative from the plugin directory. (Everything really) More information can be found by opening the extension point description. The best way to learn how to create a new probe inserter, is to look at some examples of the probes which are combined with InsectJ. But here are some points to look at: - The probe inserter has to implement the abstract "instrumentX()" methods, but it can also override init(ClassGen) and finish(). Make sure to call the super methods as well. - It is easier for the probe inserter to call the monitor methods by defining a name for every method in the interface by using the @RTInsectInvoke annotation. The probe inserter can get the invoke instructions for these methods by using the "AbstractProbeInserter.getInvokeInstruction()" and "AbstractProbeInserter.getInvokeInstructions()" methods. This is only needed for every class once, so it is best to store those instructions in a field in the init(ClassGen) method. - The probe inserters are instantiated by the instrumentation framework and need to have a public constructor without any arguments. The AbstractProbeInserter defines a constructor which takes a class object which needs to extend MonitorObject. The new probe inserter needs to call this constructor with the class object from the monitor interface it uses. This is used for type checking before the instrumentation. - The "setArguments(String)" method in AbstractProbeInserter can be overridden to let the user of the probe inserter pass arguments to it. This can be used to customize the instrumentation. The method entry and exit probe inserters use this to customize the amount of data they collect. Instrumentation The hardest part is the instrumentation self. This is done by using the BCEL library. The normal way to instrument a method is to scan the instruction list for what you want to be instrumented. When this is found, create a new instruction list object, and append the instructions you want to add to this list. And then insert it using Util.insertProbe(). We call new instruction list the probe. Every probe contains a sequence of instructions which change the stack and as the last instruction call the monitor, bringing the stack back to its original state. To call the monitor the stack should contain the monitor object instance and all the arguments which the probe inserter wants to send to the monitor. The "getMonitorObjectInstance()" method creates an instruction which pushes the monitor instance on the stack. The "getProbeId()" method gives an unique ID, which you probably want to pass to the monitor in an argument. Monitors can use this ID to identify the probe and get context information, like the method signature. If the probe inserter uses "getProbeId(String)", the String will be stored inside the class as extra context information, which can be retreaved by the monitors. The shuffling of the elements on the stack to place them in the correct order is almost an art. Many examples can be seen in the existing probe inserters. The Util class contains some methods to do some common operations, such as wrapping and unwrapping primitive values, and pushing Class objects on the stack.