Makefile Build Tools

At the terminal information is printed about processed files and any unknown files found.

mkupdate
processed: visitdataB.h visitprint.h typedefs.h visitprint.cpp visitdataA.h main.cpp visitdataC.h main.h visitdataC.cpp visitbase.h
unknown:

To make this easy I have a function which captures the first few lines of errors, and inserts the appropriate options.

Compiling the module.
$ mkerrors

id=debug
command=make
libraries=
exitstatus=0
g++ -Wall -I../misclib/ -I../visit/ -I./ -c ../visit/visitprint.cpp
g++ -Wall -I../misclib/ -I../visit/ -I./ -c ../visit/visitdataC.cpp
g++ -Wall -I../misclib/ -I../visit/ -I./ -c ./main.cpp
g++ -Wall -I../misclib/ -I../visit/ -I./ -o main visitprint.o visitdataC.o main.o

The important information to look at the is exitstatus. If this is non-zero then there is a problem.

More detailed information is stored in projcompile.txt, and is used by the CI(Continuous Integration) html reports.

Including and managing object libraries

When compiling the two places of interest are commands at the start of the compiler call and commands at the end. Library linking are commands at the end of the compiler call.

I decided on a very simple model that looks for a file called libraries and appends its contents at the end of the compiler call.

For example to include OpenGL graphics the file "libraries" contains the following line of text.
-lGLU -lGL -lglut

This approach has a per directory compiler linking options approach. This is an issue of portability as different systems have different library linkage.

To address this projlibsave projlibload projlibclear can be used to save, load and clear proj/xxx/library files. [proj/ide/projlib.sh]

For example to set up the libraries on my Mac
$ cd ${projdirectory}/ide       [ $ proj ide ]
$ projlibload librariesMacOSX.txt

Lets say I made some minor configuration changes that I want to occasionally use. Put them in a text file and call to overwrite current library files (as specified). Save your original configuration before to revert changes.

The library configuration format is the module/project directories name, a "," and then the options.

Typical examples
zpr,-framework GLUT -framework OpenGL in MacOSX
zpr,-lGLU -lGL -lglut in Linux
zpr,-lopengl32 -lglu32 -lglut32 in Cygwin

Configuring Compiler Commands

Managing compiler configurations and linkage supported. Edit xml configuration file proj/mkerrorsconfig.txt.

<command> </command> file.cpp <libraries> </libraries>
$   g++ -Wall   file01.cpp   --framework GLUT -framework OpenGL

The start and end of commands can be overridden by the command and libraries tags respectively. Each compiler option has a id which uniquely associates the command and optionally the linkage.

The presence of library file overrides the configuration file and affects only the libraries tag. e.g. set all options to include this library irrespective of their different compile options.

Some reasonable defaults for Linux were given. These can be overridden.

mkerrors   - compiles with "g++ -Wall"
mkerrors clean   - removes *.o and main .
mkerrors gdb   - compiles with "g++ -g -Wall".
mkerrors release   - compiles with
"g++ -DNDEBUG -03 -Wall".

Calling make directly

• Compiler get confused delete object code and recompile.
  $ make clean   deletes everything and is the safest.
  
• Targeted recompilation
  $ make del targ=windowscale
  to delete the object code *windowscale*.o and automatically recompile.
• $ touch file.cpp   to re-arm dependencies and then recompile.
• To change the compiler options overload the variable at the command line. First remove the previous object files
  make clean.
  Then override the CC variable.
  make CC="g++ -DNDEBUG -O3 -Wall"
make CC="g++ -g -Wall"


Advantage: know exactly what is going on. Do this before automating.

Disadvantage: can not use id's/configuration file.

Managing object compilation

If changing strategies for example compiling release code where before you were compiling debug code then delete the object files.
$ make clean

For large projects deleting all object code is time consuming because of the cost in time when you recompile the project.

In big projects for code edits near the root of the dependency tree use Corrupt Compilation.

For leaf compilation - compilation at the leafs of the dependency tree, generally use
$ mkerrors.

Corrupt Compilation

mkupdate

Consider editing code A which depends on code B. Code C also depends on code B. Now if we edit code B then code A and C will be recompiled.

For large programs the results in unnecessary recompilation of areas of code which are not used.

To correct this support for limited(or corrupt) compilation is supported. Here only the target object and its dependencies are recompiled. As long as other parts of the application are not used everything is ok.

Example
$ mkupdate - to build the Makefile
$ make - to build the object code and binaries
$ mkupdate corrupt=randomtest.o - changes the makefile to
    main: randomtest.o
Edit random.h, now we can recompile the application without other object files which depend on random.h being recompiled.
$ make - compiles randomtest.cpp only.

The advantage of what I have called corrupt compilation is that as the size of the source code increases, the compilation time need not increase. This is of course in situations where this technique is useful.

The client then uses the tool as follows. First build the application.
mkupdate
make
This ensures that all the object files exist. Now rebuild the makefile by adding the target dependency as an argument to mkupdate.
mkupdate targ.o
Freely edit targ.o's source dependencies and compile in the usual way. When finished convert back to the correct makefile with mkupdate.

In the previous example there was over 3 times gain in compilation time, and this will increase as the application grows.

The technique of corrupt compilation results in significant compilation time savings. The project can grow (independently) without effecting the ability to compile low level source files.

How my coding has improved

• More easily manage different configuration options, both in libraries and compiler commands.
• Easily change compiler strategies. This means that I can do more testing in release. My release and debug do not diverge. (CI)
• Easily share code between modules. Because this has been so successful there is a blur between what should be where as the code is automatically found and linked - a problem I am happy to have.
• Coupling with Continuous Integration the unit tests have been rewarding. Writing more testing code. Many of my projects were previously written without tests and these are still supported though CI without unit tests. e.g. Compilation breaks are quickly detected.
• Source control lets me manage file changes although I would like GIT for history currently I use subversion.
• The quality of my code has improved with the tool because sloppy practices needed to be changed. e.g. Same header file in two directories breaks the tool.

The Corrupt Compilation can lift performance as it allows the programmer to take direct control of compilation dependency. You know when to use this tool when you see files that you are not using continually being recompiled.

Different environments lead to very different wait times for building the makefile.

For example making the Makefile in the proj/graphicslib/ module takes a few minutes under Windows in Cygwin on an old P2, but takes 20s on a Pentium D with Fedora 6 os.

Why not use Makefiles to update themselves

I was unhappy with the complexity of Makefiles. In my opinion a technology should be easy to use and seeing the errors is critical.

I had a problem with the GSL makefile and it was so complicated that I could not debug it. Similarly Qt's makefile generation tool is pretty complicated too. So for now I want a simple tool that generates makefiles which are easy to see what is going wrong.

To this effect my tool explicitly writes/realizes dependency paths - uses variables simply and only a few. Effectively using a simple subset of Makefile technology and throwing the rest away.

To conclude my aim was to generate a simple makefile so that if there is a problem it can easily be seen. For example if a source file was not processed it would not appear in the makefile so the makefile generated was wrong. Then back track by looking at the error messages, back track further to the logic of the program if necessary.

For an alternative solution with Makefiles which I do not recommend see "GNU Make" 3rd edition page 33 where dependency information is included in the Makefile, pages 149+.

Program crashes and gdb

When things go wrong gdb ddd is there. Delete the old object and executable code, then recompile with -g option. Then enter a gdb session.

$ make clean
$ mkerrors gdb
generated the new code.
Start gdb, kill annoying confirm messages, load and run the program for a back trace.
$ gdb
(gdb) set confirm off
(gdb) file ./main
(gdb) r prog=34
(gdb) bt
(gdb) kill

This is generally enough to solve most of my bugs as I am an intuitional programmer I put the software in my head.

Bash Integration

This is a command line IDE(Integrated Development Environment) in a Bash OS.

proj/makefilebuildtool is compiled in release and copied to mainrelease. .bashrc loads the integrated bash code by sourcing in proj/ide/ide.sh which defines mkupdate.

[bootstrapping mkupdate] To compile the makefile build tool the bash version is used to build ${projdirectory}/makefilebuildtool. See proj/ide/MakefileUpdate.sh.

Issues

• mkupdate(C++)

  Set limits on file length names. For the hash data structure it needs fixed size allocations, so for massive projects the numbers would need to be changed - but it has been designed for this.

• mkupdatebash

  Circular dependencies are not addressed, but in practise make itself discards them.

  Commented out files are still parsed. Header #ifndef and other C++ directives are ignored so files can not be excluded from the makefile compilation with compiler directives. For example dead code
  //#include <global.h>
  sucked in and compiled 5 other irrelevant files.

  When things are set up correctly there are no problems and the makefile is generated. When things are set up incorrectly there can be logical errors in the script, circular dependencies... This program does not report the errors in a clear way, it sooner crashes.

  Each header file must be unique else find will break in the script. if [ -e $hfilename ]; then will complain about a ternary operator when expecting a binary one. This translates to the find returning two arguments so there are two file names. Both should be sent to the errors list.