Exactly how this is done depends on the device you are using and your project requirements, but the general approach when compiling applications that use a bootloader is to allocate discrete program memory space to the bootloader and application so they have their own dedicated memory. In this way the operation of one cannot affect the other. This will require that either the bootloader or the application is offset in memory. That is, the Reset and interrupt location are offset from address 0 and all program code is offset by the same amount.
On PIC18 devices, the application code is typically offset and the bootloader is linked with no offset so that it populates the Reset and interrupt code locations. The bootloader Reset and interrupt code merely contains code which redirects control to the real Reset and interrupt code defined by the application and which is offset.
On mid-range devices, this is not normally possible to perform when interrupts are being used. Consider offsetting all of the bootloader with the exception of the code associated with Reset, which must always be defined by the bootloader. The application code can define the code linked at the interrupt location. The bootloader will need to remap any application code that attempts to overwrite the Reset code defined by the bootloader.
The option -mcodeoffset, (see Codeoffset Option), allows the
program code (Reset and vectors included) to be moved by a specified amount for devices
that do not use the VIC module. The option also restricts the program from using any
program memory from address 0 (Reset vector) to the offset address. Always check the map
file (see Map Files), to
ensure that nothing remains in reserved areas.
For devices with the VIC module (even those operating in legacy mode) you will need to adjust the IVTBASE register to move the hardware vector locations.
The contents of the HEX file for the bootloader can be merged with the code of the application by adding the HEX file as a project file, either on the command line, or in MPLAB X IDE. This results in a single HEX file that contains the bootloader and application code in the one image. HEX files are merged by the HEXMATE application (see Hexmate). Check for warnings from this application about overlap, which can indicate that memory is in use by both bootloader and the downloadable application.