IAR support

[praqma-thi]

Support for IAR compilers

Trello card from old roadmap with a few more details that can be useful.

[buep]

Roland Stahn mentionson the Trello card that he have added some example IAR Linker MAP files to this issue in the Jenkins JIRA: https://issues.jenkins-ci.org/browse/JENKINS-19840

[buep]

See also the relevant comments and supplied examples from Roland Stahn in JENKINS-19840 (which is now closed as we only use GHI going forward).

[kvdz]

HI, i would like to make this, actually is have started already. https://github.com/kvdz/memory-map-plugin
but what is adviced?

look for each for ".text", ".noinit", ".bss"and ".data"?
Or look for the summary of the compilation:

Module                  ro code  ro data  rw data  rw data
                                                    (abs)
---------                -------  -------  -------  -------

---

Grand Total:             22 857    2 487    3 469      230

specially the Grand Total: ..... line.
because of proprietary software the content has been excluded.

The numbers are an example how much flash is used for code and data, same counts for RAM use

[buep]

Hi @kvdz

Great, we're happy to take in contributions and supply you with some answers etc.

Regarding your question above we would prefer if find the different .text, .bss etc so they can be used for making graphs. You need to implement a new parser in src/main/java/net/praqma/jenkins/memorymap/parser/ like the one for TI and GCC.
I haven't programmed any of it, but it should mostly be copy past and then fixing the regular expressions that looks for the values and adjusting a few other things.

For testing you should supply as minimum some example output file you can use in testing, so maybe from a non proprietary example?

Please note that to get your changes release you should make the pull request on this repository, not the jenkins-ci your fork i based on. But that is simply a matter of some git gymnastics.

Let's keep in touch on this issue.

[kvdz]

The reason i am asking is because i am not confident that it will work for IAR map file.
why because the illustrations in here https://github.com/Praqma/memory-map-plugin/blob/master/docs/ResearchNotes.md are not always true for IAR.

you could have 2 programs (bootloader, main program) bootloader can ben in the upper space of in the lowest space of the flash. The main program and the bootloader can have the same vector table or have different onces.

also the output from the IAR compiler (map file) is set up that it is grouped by sections of flash. into absolute sections and relative sections and what this sections contains. i don't see any of this in the TI of gcc test map file. By TI and gcc they are grouped by memory type.

[kvdz]

example some parts are removed:

---

*** PLACEMENT SUMMARY

---

"A1": place at 0x10001fe0 { section .noinit };
"A2": place at start of [0x00000000-0x000002ff] { ro section .intvec };
"A3": place at start of [0x00000300-0x0000cfff] { ro section .cstartup };
"P1": place in [from 0x00000300 to 0x0000cfff] { ro };
"P2": place in [from 0x000002fc to 0x000002ff] { ro section .crp };
"P3": place in [from 0x10000000 to 0x10001edf] { rw, block CSTACK, block HEAP };

Section Kind Address Size Object

---

"A2": 0xc0
.intvec ro code 0x00000000 0xc0 LPC11xx_IAR.o [1]
- 0x000000c0 0xc0

"P2": 0x4
.crp const 0x000002fc 0x4 GeneralAPI.o [1]
- 0x00000300 0x4

"A3": 0x10
.cstartup ro code 0x00000300 0x10 LPC11xx_IAR.o [1]
- 0x00000310 0x10

"P1": 0x622c
.text ro code 0x00000310 0xdd8 xxxx.o [1]
.rodata const 0x00001f6a 0x2 xxxx.o [1]
.text ro code 0x00001fb8 0x3b0 main.o [1]
.rodata const 0x00005a84 0x468 xxxx.o [1]
.iar.init_table const 0x00006390 0x24 - Linker created -
.text ro code 0x000063f0 0x14 LPC11xx_IAR.o [1]
.text ro code 0x00006404 0xc cstartup_M.o [5]
.rodata const 0x00006410 0x0 zero_init3.o [5]
.rodata const 0x00006410 0x0 copy_init3.o [5]
Initializer bytes ro data 0x00006410 0x12c (used: 0x128)
- 0x0000653c 0x622c

Absolute sections, part 1 of 34: 0x4
.noinit uninit 0x10000000 0x4 Bootloader_LPC11xx.o [1]
- 0x10000004 0x4

"P3", part 1 of 3: 0x12c
P3 s0 0x10000008 0x128
.data inited 0x1000011c 0x4 Bootloader_LPC11xx.o [1]
.data inited 0x10000120 0x4 GeneralAPI.o [1]
- 0x10000130 0x12c

"P3", part 2 of 3: 0x450
.bss zero 0x10000130 0xfc main.o [1]
.bss zero 0x1000022c 0xac main.o [1]
.bss zero 0x100004b0 0x20 main.o [1]
.bss zero 0x100004d0 0x20 main.o [1]
.bss zero 0x100004f0 0x20 main.o [1]
.bss zero 0x10000524 0x10 GeneralAPI.o [1]
.bss zero 0x10000534 0x10 main.o [1]
__DLIB_PERTHREAD zero 0x10000564 0x4 errno.o [3]

                        - 0x1000057e   0x44e

"P3", part 3 of 3: 0x800
CSTACK 0x10000580 0x800
CSTACK uninit 0x10000580 0x800
- 0x10000d80 0x800

[rstahn]

Nice to read about progress on this feature! I am happy to provide even more map files to be tested: EWARM_MAP2.ZIP
This contains MAP output of IAR EWARM v7.60.1 and the quite new v8.20.1.

I also added the associated linker configuration file for each map file, since I just learned that the memory map plugin evaluates this as well. My examples include three different STM32 devices (STM32F0, STM32L1, STM32L1-X). Especially the STM32L1-X is interesting, as its memory layout contains two flash banks (192kB each) with a gap in between!

Is the new IAR parser already able to evaluate the linker config files?

[rstahn]

BTW, since this issue is called "IAR support". I think we need to differentiate between the various editions of IAR compilers for different platforms. Most users are probably interested in parsing map files for the widespread ARM MCUs. This is what IAR calls "EWARM" (Embedded Workbench for ARM). As we already learned from my example files, linker output is different for other IAR compilers (STM8, AVR, etc.). I would like to suggest to concentrate on EWARM support first.

[kvdz]

@rstahn i agree on seperation of the issue into IAR EWARM/ IAR AVR/ IAR STM8. the only tested versions with this parser is a LPC 11xx, 17xx using IAR V6.4.4 so newer versions are welcome. but i don't have the rights to change issues here. i will test the zip you posted here to see if there is room for improvements.

[kvdz]

@rstahn The most parts like .bss, .rodata, .text, .data and .noinit are all good on any version (V7.60.1 and V8.20.1).

only i saw something on EWARM7601_STM32L151VD-X.map
i saw this __DLIB_PERTHREAD inited 0x2000339c 0x4 errno.o [3]
where is __DLIB_PERTHREAD stored? is it in rom? ram?

[rstahn]

where is __DLIB_PERTHREAD stored? is it in rom? ram?

Apparently, it is a special section for "DLIB multi-threaded support":
http://supp.iar.com/FilesPublic/UPDINFO/005691/arm/doc/infocenter/DLIBThreadSupport.html

I could find only limited documentation on this section within the IAR manuals:

__DLIB_PERTHREAD
Description: Holds variables that contain static states for DLIB modules.
Memory placement: This section is automatically placed correctly; you must not change its placement.

The DLIB library supports TLS memory areas for two types of threads: the main thread
(the main function including the system startup and exit code) and secondary threads.
The main thread’s TLS memory area:

• Is automatically created and initialized by your application’s startup sequence
• Is automatically destructed by the application’s destruct sequence
• Is located in the section __DLIB_PERTHREAD
• Exists also for non-threaded applications.

Each secondary thread’s TLS memory area:

• Must be manually created and initialized
• Must be manually destructed
• Is located in a manually allocated memory area.

From the address in the MAP file I can conclude, that it is placed into RAM. But I can only guess, that this is always the case.

[kvdz]

@rstahn oke thank you.

[kvdz]

@buep how does the configuration works? to me it looks really messy and not really readable. What should it do in these functions:"parseConfigFile, parseMapFile" ?

[kvdz]

@rstahn i have added all ARM AVR and STM8 memory sections so no rename needed on tickets.

[rstahn]

i have added all ARM AVR and STM8 memory sections so no rename needed on tickets.

Thats great! Thank you.
IAR offers a bewildering variety of editions though. Have a look at their website and click on "architecture" to get a quick impression. I have no idea whether these editions use different linkers (and MAP files). I really doubt that they should all be covered with this single issue. Luckily that decision is not up to me. All editions I can provide example files for are supported now. :)

[ben-edna]

Really looking forward to IAR support for the Memory map plugin, is there anything that I can do to help the merging of PR #43 ? Is it easy to create an .hpi for the branch that we can test the plugin with IAR support?

[praqma-thi]

Hey folks,

Sorry if things have been quiet from our side. We've had a hectic spring and now everyone's buggered off for summer holidays. 🌴

I'll see if I can help you out and get the ball rolling for when @MadsNielsen gets back from holiday. I'll look into @kvdz's changes and questions, and setting up a HPI build for the PR.

Expect some info soon. ✌️

[praqma-thi]

I pulled down @kvdz's changes, rebased them onto master and tinkered away a bit to get it building.
My changes are on the iar-pr-1 branch.

The project still uses Maven, so to build an HPI you can run mvn clean package, which creates target/memory-map.hpi. You'll need Maven and the JDK installed, or use the maven:3.5.3-jdk-8 Docker image, which we use for our CI builds.

[praqma-thi]

As for the details regarding the parsing methods, I only have a vague idea.
I think I'll need to chat with @MadsNielsen next week. 😅

I'll take a shot at explaining them anyway, but take what I say with a grain of salt. I can do Java, but memory maps aren't really my thing. 😉

From what I understand that's where the RegEx are actually used to build up a list of MemoryMapConfigMemoryItem objects, which represent memory blocks and are used to publish the results and draw the graphs.

The parseConfigFile method parses the linker files (ld, cmd, (icf?)) and creates a MemoryMapConfigMemory object, which is basically a list of MemoryMapConfigMemoryItem objects. The method adds an item to the list for each memory block and any information it can parse from it, generally the start address and length.

The parseMapFile is called afterward, getting the MemoryMapConfigMemory object that parseConfigFile made before. This parses the actual map files and fills up the items in that list with any extra information it can parse.

For example, the parseConfigFile would parse the linker file and find which blocks of memory were reserved and with what length, and parseMapFile would fill in the gaps by recording how much of the block was used/unused.

Hope this makes a shred of sense. 🙂
If not, we'll have to wait until I can go through it with @MadsNielsen.

[rstahn]

Any progress regarding this topic?

[ben-edna]

Is there anyone that has this plugin (with IAR support) working in a pipeline and can provide a demo snippet, I'm not sure how to use it. But I'm still very much open into testing it, we have a lot of different IAR versions and platforms. Currently trying to integrate it into a 7.80.4 for ARM project.

I created a hpi and installed it on our Jenkins, but I can't seem to figure out how to use it correctly from a pipeline, keep getting No match found for program memory named ..., and tried everything.

[marc-pe]

Since yesterday, I'm stuck at the same problem: No match found for program memory named .... Tried things like const, CONST, Data, DATA, data.
What Linker command file did you use?
I used your hpi, as I compiled my own but it didn't have a "Publish Memory Map post-build action" made available

[ben-edna]

I tried both the icf and the map file, since in IAR everything should be parseable from the map file.
Next to that I tried all kinds of entires for all fields such as Code, Const, .bss, .text, text, EWARM_CONST, EWARM

[praqma-thi]

Hey everyone,

We're looking at fixing and merging in the pull request, then making a formal release that supports IAR. To help with that, we could really use some more IAR files to write tests against, as we currently have no way of verifying if the parser's working. 😕

Would it be possible to provide some test files you'd like being able to parse using the plugin? If possible, specify which version of IAR you're using as well. Even the output from hello world projects would be greatly appreciated.

Thanks!

[ben-edna]

Great! 👍

Below some example map files (as .txt). If you need more examples I'm happy to provide them, but you can also download IAR for a specific architecture and on the start page go to the Information center and download example projects.
The exact linker version for the file output is mentioned in the header of the map file

EWSTM8 2.20

IAR-EWSTM8-2.20.txt

EWARM 7.40

IAR-EWARM-7.40.txt

EWARM 7.80.4

IAR-EWARM-7.80.txt

EW8051 9.20.1

IAR-EW8051-9.20.txt

[marc-pe]

Glad to hear, that you're on fixing it.

I attached some map files (as .txt, as .map is not supported to upload).
If you want me to test a prerelease version of your plugin, I 'd be glad to do so with some more map files, which I can't publish as they contain confidential information.

IAR-EWARM-7.80_File2.txt
IAR-EWARM-7.80_File1.txt

[DaMutz]

I can generate more map files if you need them. I am very interesting in this feature and the plugin in general. When is this feature planned to release? I can also contribute with unit-tests or parser implementation.

[OEHC]

This is the next issue I will work on. I hope to get to it within two weeks.

[OEHC]

The current implementation on the iar-pr-1 branch does not take the changed format of the linker files into account and is therefore not able to correctly match the different things to be graphed. Could you provide a few example linker files as well, so we can get this fixed?

[spoorcc]

Below some linker files as text files (remove the .txt). Note that the 8051 linker files have a .xcl extension and the other .icf.

• ew8051_9.20.1_linker_file_lnk51ew_P80C31BA.xcl.txt
• ewarm_7.40.7_linker_file_stm32f042x6.icf.txt
• ewarm_7.80.4_linker_file_LPC834M101.icf.txt
• ewstm8_2.20.1_linker_file_lnkstlux285a.icf.txt

[rstahn]

I can provide additional linker configuration and map files for EWARM 7.60.1:

• EWARM_7.60.1_linker_stm32f091xC.map.txt
• EWARM_7.60.1_linker_stm32f091xC.icf.txt
• EWARM_7.60.1_linker_stm32f303xE.map.txt
• EWARM_7.60.1_linker_stm32f303xE.icf.txt
• EWARM_7.60.1_linker_stm32l152xE.map.txt
• EWARM_7.60.1_linker_stm32l152xE.icf.txt

I did also check the linker files for EWARM 8.22.3 - they are similar to EWARM 7.60.1. It looks like there were no changes to the ICF file format with the latest major version.

[rstahn]

I can also provide some additional ICF files for larger MCU's with multiple SRAM and FLASH banks:

• EWARM_8.22.3_linker_stm32f479xI.icf.txt
• EWARM_8.22.3_linker_stm32f777xI.icf.txt
• EWARM_8.22.3_linker_stm32h743xI.icf.txt

I do not have a map file at hand for these MCU's though.

[rstahn]

Here is another interesting special case:

• EWARM_7.60.1_linker_stm32l15xVD-X.map.txt
• EWARM_7.60.1_linker_stm32l15xVD-X.icf.txt

This controller features 384kB of FLASH memory in two banks with a "gap" in address space between the banks. Bank 1 covers 0x08000000 - 0x0802FFFF and Bank 2 covers 0x08030000 - 0x0805FFFF. The address range 0x08030000 - 0x0803FFFF inbetween has no flash memory.

[ben-edna]

@OEHC Is there a branch in a fork we can try out, even if it is not finished yet? Would love to help in testing it. If there is anything else you need, please let me know.

[praqma-thi]

The relevant changes should be on the iar-pr-1 branch.

I'm planning on picking this up again on Friday, as Oliver went on leave to finish his thesis. I believe I was adding some tests when I last left off, though I seem to have done an excellent job of actually pushing my changes. 😓

Regardless, I'll see if I can get those tests up and running and share some more detailed notes here.

[ben-edna]

@praqma-thi Thanks for the update! Let me know if I can contribute in any way. Really appreciating the work 👍

[praqma-thi]

Fruitful afternoon digging through old Java code, I must say.

The current IAR parser is a placeholder copy of the Texas Instrument parser. Before jumping into writing the IAR parser, I tried to figure out how implementations of AbstractMemoryMapParser actually work. I did this mostly by taking a look at the existing Texas Instruments parser implementation. This turned out to be a poor decision, as the TI parser only works because of pure luck.

When you configure your graph to record, for example, .bss, it should use the correct regex to find .bss entries in the map file. The logic which ties these things together is found in the MemoryMapConfigFileParserDelegate. Looking at that code, you'll quickly notice that, if the section you're looking for (.bss) doesn't already have a pattern registered, it will return a default pattern that uses the section name.

Looking through the code and running a few tests, I've concluded that it always uses the default pattern, which luckily works for both Gcc and Texas Instruments, and that the other fancy patterns and Java code in those parsers is effectively pointless.

After a long exhale, I decided to keep things simple in the IAR parser, simplify the whole "which pattern to use for which section" to a simple map. e.g.:

Map<String, String> sectionPatterns = [
    ".text": "\\.text\\s+\\S+\\s+\\S+\\s+\\S+\\s+(\\S+)",
    ".intvec": "\\.intvec\\s+(const|ro\\scode)\\s+\\S+\\s+(\\S+)",
    ...
]

And then have the parseConfigFile and parseMapFile methods use that map to use the correct patterns for parsing. If I like how that looks when I'm done, I'll probably go back and clean up the Gcc and TI parsers as well.

That approach doesn't cover different versions of the compiler yet (EWARM, STM8, AVR). Either we can try detecting the right version and use the appropriate patterns, or a second drop-down to pick the version could work as well.

Regardless, I'll let this simmer over the weekend, and get down to implementing this either Thursday or Friday.

[praqma-thi]

Pushed some WIP changes to the iar-pr-1 branch.
Added a placeholder test, moved all the patterns to map and started working on the actual parsing logic.
I'll have to take another look at what I actually have to parse and where to put the data, though I don't think that'll be too difficult.
If we're lucky, I might get that test functional and green when I come back to this on Thursday.

[praqma-thi]

I was plagued by a weird runtime error, where Jenkins complains the MemoryMapRecorder class doesn't have a Descriptor, but it does. I took my builds to the build server where the tests run without issue.

So now the parser finds and parses entries, but results don't match expectations.
It looks like it's not correctly adding up the results for each section.

So parsing .bss on the following, it ends up with 0x70, rather than the sum, which should be 0xc4.

"P2", part 3 of 3:                          0xc4
  .bss               zero     0x20001008    0x70  HAL_Core.o [1]
  .bss               zero     0x20001078    0x54  xxwritebuffered.o [3]
                            - 0x200010cc    0xc4

Regardless, it's progress. More to follow.

[gjabouley-invn]

Hello @praqma-thi , any update on IAR ARM (EWARM) support readiness for this plugin?
Will it address both v7 & v8?

[praqma-thi]

Progress is slow, as I can only rarely chip away at it. I'll see if we can somehow address this, either by reviving Praqma's open source department or putting the plugin up for adoption.

Some progress notes:
I spent some time digging into why the parser isn't returning the correct values.
So the parser goes through the map file, and looks for entries you want recorded.
However, I found out it the parser doesn't record multiple entries with the same name (e.g. .bss).
This is due to an @OverRide of the list's add method. I'll have to do some digging on why this was added in the first place. Removing the override doesn't break any tests for the existing parsers, so I'll tentatively do so.

It still doesn't return the correct values (now returning the value of the last entry, rather than the sum of all), which'll require a little more digging.