Gen 2, Summarize informations #68
Comments
|
Hi, the drivercards (gen1) I saw until now starts with 0x76 0x01. If you want to restart this project I think that we have to identity the type I also make some research on and move it .net Core and make some clean ups. |
|
Hi, I probably need to implement partial support for the version 2 of the specification. Can someone share with me at least a few gen2 |
|
I don't work at company that works with Tachographs anymore. Pretty sure they have gen2 files but it's not possible to share them because they contain customer data and GDPR all that. |
76 01/21, 76 02/22, 76 03/23 etc... are identifiers of data blocks in the ddd file from the tachograph (01 - 1gen, 21 - 2 gen). In the ddd file from the driver card, it may start with 00 02. This is the EF ICC identifier. |
|
What file is the file you used to get those pictures? We as a company are also busy trying to decipher gen 2, and when I figure it out I plan on implementing it on the Jugglingcat lib so nobody ever has to do this work again hopefully for the coming years. |
|
Yeah but thats the problem. I cant find a similar spec like on the first generation anywhere. I used ECE/TRANS/SC.1/2006 for the first generation, and then appendix VII gives a pretty clear documentation on what the file looks like when downloaded. But there isnt a similar document to be found for the second generation as far as i can tell |
|
I can upload some test data that are free of any identifying information. |
|
That would probably be very helpfull! |
|
Here is sample data. Be advised that the certificates used are test certificates, so they will not properly validate. The dataset contains driver cards and transfers from a vehicle unit. None of the data contain real persons, so there are no privacy concerns. |
|
Wow thanks man! great help. To give some context on what i'm doing: I'm currently working for a small company where we create devices to remote download the tachographs in trucks. I've been asked to check how we can validate the downloaded files. We were using the @jugglingcats lib for testing and finding out how it all comes together. It worked great for GEN 1. Now i've been asked to look into GEN 2. For research purposes i'm now creating some simple python code to pull the data from the gen 1 and 2 files, gen 1 works, but now still gen 2. When that is done, I need to find out how to validate it all, but thats the real part I can't make heads or tails of. The getting data from the GEN 2 file is already succeeding bit by bit. I find the record arrays actually pretty handy dandy. |
|
Well, you can run through all of the data structures and verify plausibility, to one degree or another, but generally it is enough to verify the signatures to have full confidence in the download not having been corrupted or tampered with. |
|
Yeah, in the Netherlands we say "je slaat de spijker op zijn kop". Which roughly translates to "you hit the nail on its head with your comment". Verifying the signatures is exactly the part I don't understand. But I'll get to it someday, and when I do I want to update the @jugglingcats library so nobody ever has to do this sheit again. |
See #42, there I linked latest doc at the time - https://eur-lex.europa.eu/eli/reg_impl/2016/799/2018-04-17 It's not exactly that readable but info is there you just need decypher it in your mind 😂 back then I went over it many times and got pretty good understanding but it was many many years ago. |
|
Looks like this https://eur-lex.europa.eu/eli/reg_impl/2016/799/2023-08-21 (interestingly date in future :D) is the most newest one now. |
|
@mpi-wl |
0x7621 is "Transfer data overview structure generation 2, version 1 (TREP 21 Hex)" - here is a direct link to the information: CELEX:02016R0799-20230821 (page 343 if it doesn't open on its own) It will look something like this: |
|
Hi @graealex, thanks for your response. I must have missed this piece in the documentation, which isn't difficult since it is 600 pages long lol. My problem is that I can't read Gen 2 files manually. I have a couple of files, for example, gen 1 which starts with But when I've opened the second file which starts with Am I doing something wrong? On what should I focus on? |
|
Files with sections starting with 76 01 or 21, 76 02 or 22, 76 03 or 23... etc. are tachograph readings. |
|
As @mpi-wl correctly wrote, 0x7621 indicates that this is a download directly from a VU (vehicle unit - the actual device memory). All these files come as .DDD files, but can be just a card or from a VU. And yes, there is Gen 1, Gen 2 V1 and Gen 2 V2. And for VU downloads, the format has actually changed quite a bit between Gen 1 and Gen 2, notably because every block is now a record array, and never just the block itself. |
|
I have written all the code for the company I work for which parses all three gens into usefull data objects so we can just extract everything. I happen to know a little bit about it now, but I don't really understand your question? Yes there are multiple generations, mainly gen 1 and gen 2, and a latter gen 2.1 where gen 2.1 mostly just adds GNSS data to different fields. Could you summarize what your question is? |
|
I'm ashamed to admit but I was sure that the files with the extension .ddd are only files from the driver's card. For a few days, I'm trying to understand the structure of these files, file types, manual reading, etc. |
|
@Camillus83 There is no shame in not knowing that, because it's not like any of the tachograph stuff is particularly well documented. Generally there is a download protocol ("Appendix 7 DATA DOWNLOADING PROTOCOLS" in the EU spec) where a device requests blocks from the VU to download, and one such answer is the 0x7621 "Transfer data overview structure generation 2, version 1 (TREP 21 Hex)" block. These answer blocks then get written into a DDD file. |
|
Ah okay, No, as said before, Both the driver card or the tachograph mass memory can be downloaded via the tachopgrahs interface. Both are called 'ddd' files. But they are completely different. The Driver card looks exactly like the structure described in the documents. And is basically just all card data in a specified order. LIke any card file system. Gen 1 memory files are very different from Gen 2. Where gen 1 is just bytes in a specified order and gen 2 consists of so called 'record arrays' where each record array first specifies the type, length and amount of fields giving you the chance to see the length of a section before actually looking at the data. The data then is about the same as it is in Gen 1. But there are some differents between all data in gen 1 and gen 2, so be mindfull of that. Gen 2 mostly has more data for specific fields then gen1. Gen 2.1 is about the same as gen 2 except for added GNSS data. If you need more detailed info I can help. Someday I will upload all my code giving everyone an easy to use library to parse gen 1, 2 and 2.1 but for now im under NDA. I have to rebuild it in a different way to do this, but yeah. Time and all. |
|
As I said before, thank you for the explanation. Now I am able to read the files.
It seems that the generation changes and the rebuilding of the structure made a lot of mess in the existing solutions and in the creation of a compatibility solution with all generations. |
|
Hi Guys, it's again me. After couple of weeks I've finally understood and built parser for Driver Cards. Now I started working on VU files, and again have some problems and misunderstandings. First main issue was when I was trying to read VuActivities block from VU Gen1 file.
Block structure from my file looks like:
But when I've opened that file in HexEditor I saw that there are many more blocks which looks exactly the same like VuCardIWRecord and have the same prefix as 00 01. It seems to me that I don't understand or wrongly reads file, because later other blocks after parsing have meaningless data. |
|
well I would have to look it up in the code and documentation again, its been a while for me. But vehicle unit card iw record stands for "insertion and withdrawal" record if i remember correctly. And those can be multiple. hense the number of iw records uint16_t there. You cant deduce the data of gen 2 files by just looking at the hex. You will see that all data records look more or less the same, as the whole idea of gen 2 and up files are made using the same first few bytes, indicating the type, length of data and amount of data blocks. Its only after the first few bytes that it is different. So if you just look at the hex and say 'hey, i see a lot the same', yeah that's the idea. |
|
According to the first part of message, the data blocks are built with noOfRecords, then recordsData(noOfRecords*recordData). According to the second part of your message, in DriverCard i've recognized generations based on the size etc. But In VU it's probably kind easier. |
|
Can you share the data you are trying to parse, or is it a privacy concern? |
|
Unfortunately, it's confidential data from the customer :/ |
|
CVCs ("Card Verifiable Certificates") are defined in ISO/IEC 7816-8. "Certificate verification and unwrapping" is described in detail in ISO/IEC 9796-2. Unless you find a prior implementation somewhere, I don't think you have an easy way around simply buying these specs, especially the latter. The document from which you are citing only gives a very rough overview, basically only half a page describing the method. Edit: Here is an old version of the aforementioned document free of charge on Internet Archive: https://archive.org/details/gov.in.is.iso.iec.9796.2.2002 |
|
@graealex |
|
Well, while standardization is a good thing, the fact that nearly all of these documents are behind a paywall makes it hard again. Also when you buy a document, your name will be embedded in the document, so that's why no one is sharing them. Besides the legal aspect, obviously. |
|
I read the document you uploaded and found the answer to how to verify the signature. It is explained quite clearly. Once again, a big thank to you. Nevertheless, I am still having trouble with verification. Which is an Signature 39871213657556712785743798399678659729728462930070485080026276996237931233957214655329281058127998080188609422467155620875643995926287856704772605574419436962902653026391993371019986085376311278191228766790165694547487105168431139174600037348108043676741863493392549357225865748487737088872217765342447831663 Public expotent: 65537 So with such large numbers, I don't know if there isn't some error along the way because it is Anyway, digs into it further and maybe something will come up. |
|
Take a look at these implementations: That's the most accessible example I could find, and implements all three schemes. |
|
Hello, great work and it has been of great help to me. I have problems being able to correctly translate the coordinate value read from a file. 3bytes for each Lat and Lon, I convert them to decimal but that's where I stop because I don't know how to convert that decimal into the coordinate value according to the information in the documentation. Can you Help me? Example: GNSSPlaceGeoCoordinates Lat: 41116 /// Lon: 16776023 - Hex: Latitude - 00a09c /// Longitude - fffb57
|
Latitude seems plausible. You just divide by 1000, so 41116 becomes 41.116 - that could be Greece, Italy or Spain. The other one is negative, represented as One's Complement. If you invert the bits (XOR with 0xFFFFFF), you get 0x4A8, or decimal 1192. Divide by 1000 again, gives you -1.192. Which would be here, a road in eastern Spain. I could be wrong though, as the three MM.M denote the minutes, which isn't exactly just a decimal fraction. So maybe check that multiple coordinates are plausible. |
|
Thanks @graealex , But we need extract first sign, and i don't know how. First i convert 3bytes from latitude to decimal, but in longitude how i detect if the decimal negative? you know any function in python to do this? |
I thought it was clear. If the first (MSB) bit is one instead of zero, then it's a negative number, and you need to invert the bits. Please read the article on One's Complement which I linked. That's literally how all signed numbers work in computers, with the exception that here you only have 24 bits instead of the typical 16 bit short or 32 bit int types. Another shortcut would be to shift left 8 bits, then use normal signed int logic, and then shift right 8 bits again. |
|
@aRRoYo4 Are you sure you read GeoCoordinates correctly? Could you paste the entire GNSSAccumulatedDrivingRecord? |
But the provided data returned perfectly fine and plausible coordinates... |
|
I haven't been to Spain, but it doesn't look like a truck road.
|
|
@mpi-wl Hi, we develope the block data of Places Stored. This data have two parameters with coordinates when you init or ends your worktime or change country/regions. |
|
Is the country code 0F in PlaceRecord that parses? |
|
###VU All places stored### If i'm not wrong, in Lon binary the 4 first bit are de sign negative? or the 8 first bits. I convert the integer to binary and adding right 8 zeros to detect if is positive o negative sign, but i don't if this is the correct way to translate correctly the coordinates and the sign. print(bin(int.from_bytes(gnssPlaceGeoCoordinatesLat,'big'))[2:].rjust(8, '0')) |
|
@aRRoYo4 In Python - where
And no, it is only the MSB (most significant bit) that indicates the sign. If the bit is 1, then negative. That gets checked with "AND 0x800000" which equates to "1000 0000 0000 0000 0000 0000" in binary. |
|
Thanks @graealex |
|
Ok, indeed the geoCoordinates seem correct. In my opinion Longitude: FF FB 57 -> -1193 -> -119.3 because longitude is encoded as a multiple (factor 10)... -> -1 19.3 (DDD MM.M) The location is here |
|
You're right about it being -1193 instead of -1192, because I did One's Complement, when it is in fact Two's Complement. But not so sure about the rest. Although both coordinate sets are in the province/region of Aragon. But yours seems more likely. How did you calculate the coordinate from the decimal/integer? |
|
I don't know if I understand your question correctly. Sorry if that's not what you're asking. We use three formats for recording geo coordinates:
After dividing the latitude / longitude value (from the geoCoordinates section) by 10, we get degrees and decimal minutes: DDMM.M / DDDMM.M
latitude: 41116/10 = 4111.6 -> 41 (DD) 11.6 (MM.M) This format can be pasted into the search box on Google Maps:
However, if we want to convert it to decimal degrees (e.g. to display items using Leaflet), we should do the following:
|
|
I've never checked it - I didn't ask the driver where he was at a given moment, but according to the charts everything is correct.
|
|
@mpi-wl Thanks for the explanation. This made it really clear. |
|
You need to extract the three bytes and OR them together. As you had other values also extracted from your data, I assumed that was clear. So:
|
|
Hi, i just started working on a tachograph reader and have a question. For the 0x7631 identifier, if I use the same code as the 0x7621 one I get errors, but if I read 8 bytes before, comes no trouble. I can't find documentation about those 8 bytes. Thanks. |
|
Greetings everyone. Thanks in advance. |
|
I had to finally deduct this (PHP code): while ($reader->readableBytes()) {
$ef = $reader->readUInt16();
$state = $reader->readUInt8();
$size = $reader->readUInt16();
$decoder = match ($state) {
0 => Generation1::class,
2 => Generation2::class,
};
}The
In a Gen1 file, the 0 state will always be present, as well as Gen2v1 and Gen2v2 files.
I'm doing my developments following this architecture, it seems to work well for now. |
|
Hi @mpi-wl , what library are you using to generate the graphs in the image you posted? I'm trying to find a library to be able to create daily graphs with data on vehicle speeds, fuel consumption and tachograph status, but I can't find one that convinces me. |
|
Hi @aRRoYo4 In this demo https://esmreader2.tachosfera.eu I did it in an inefficient way. This library is very good. In a large program for analyzing drivers' working time, my developers did "wonders" with this library.
|
|
thanks @mpi-wl |
Hello
I want to summarize and consult all information which I need to write update to gen2.
-The structures of bytes for the segments is the same like in gen1.
-In the segment is only one change, in new generation type can be also 0x02(content) and 0x03(sygnature).
-On the one card might be also types of gen1 and gen2
On the beggining I thought that I can compare length of data "application_Identification" becouse it is exacly 10 byte in gen1(pic.2) and 15 byte in gen2(pic.3) . After select generation, choosen config. But I found information that in the card can be mixed information (gen1 and gen2 segments).
*"segment" mean one block of data like "application_Identification"
*@Dilemma725 wrote a program which to be able to check wchich generation card is. In his code he checking first and second byte of all card byte data.
-if first 0x76 and second 0x06 - G1
-if first 0x00 and second 0x02 - G2
Why? First block of data is the same for both generation(pic1)
My question is:
Did the best way do update this application to generation 2 is getting each segment one by one and checking his type?
Originally posted by @Phoryn in #42 (comment)
The text was updated successfully, but these errors were encountered: