-
Notifications
You must be signed in to change notification settings - Fork 260
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HolgerAusB ippen.media-1 adds 50+ new newspaper sites by fingerprint #928
Merged
j0k3r
merged 9 commits into
fivefilters:master
from
HolgerAusB:HolgerAusB-ippen.media-1
Mar 11, 2022
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
39c7a49
initial upload empty file
HolgerAusB 2253323
how-to comments, still no code
HolgerAusB 6c85798
The Code
HolgerAusB 0455960
some test sites, more to come
HolgerAusB b5112c8
Update .ippen.media.txt
HolgerAusB c501dd4
Update .ippen.media.txt
HolgerAusB e6ad235
Update .ippen.media.txt
HolgerAusB a3f3172
ready to PR
HolgerAusB 5c249c3
Update .ippen.media.txt
HolgerAusB File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| # Author: HolgerAusB | ||
| # | ||
| # ippen.media is a German newspaper network that operates about 60+ websites | ||
| # with (local) news and magazines, all or most of the newspaper sites with the same CMS. | ||
| # magazine sites are not working yet. | ||
| # | ||
| #========================== | ||
| # | ||
| # If you host fultext-rss/wallabag on your own servers you may | ||
| # a) | ||
| # copy this file and name it for your domains e.g.: | ||
| # custom/hanauer.de.txt | ||
| # custom/ruhr24.de.txt | ||
| # | ||
| # or | ||
| # b) | ||
| # for fulltext-rss: add a fingerprint to the array in your custom_config.php: | ||
| # | ||
| # $options->fingerprints = array( | ||
| # //Ippen Media Newspaper Sites | ||
| # 'var ippenErr = [], ippenPrevEH' => array('hostname'=>'fingerprint.ippen.media', 'head'=>true), | ||
| # // Posterous | ||
| # ... | ||
| # ); | ||
| # | ||
| #========================== | ||
| # | ||
| # to get the source-feed add 'rssfeed.rdf' to the category-URL e.g. | ||
| # https://www.fr.de/hessen/rssfeed.rdf | ||
|
|
||
|
|
||
| # strip author box and social media box | ||
| # these boxes sometimes prevented main picure to show up | ||
| strip_id_or_class: idjs-simpletab-nav-item | ||
| strip_id_or_class: idjs-simpletab-content-close | ||
| strip_id_or_class: id-AuthorList | ||
| strip_id_or_class: id-Article-content-item-copyright | ||
|
|
||
| # strip adverts | ||
| strip_id_or_class: id-Article-content-item.id-Article-advert.id-Article-advert--ad3 | ||
| strip_id_or_class: id-Article-advert.id-Article-content-item | ||
| strip_id_or_class: id-Article-content-item.id-Article-advert | ||
| strip_id_or_class: id-Article-advert--ad3 | ||
| strip_id_or_class: id-Article-advert | ||
|
|
||
| tidy: yes | ||
| prune: yes | ||
|
|
||
| test_url: https://www.az-online.de/uelzen/stadt-uelzen/mehr-gaeste-in-der-uelzener-stadthalle-als-erlaubt-91282522.html | ||
| test_url: https://www.bgland24.de/bgland/region-berchtesgaden/bischofswiesen-ort28409/schoenau-am-koenigssee-wieder-grosses-interesse-bei-fit-durch-unser-gmoa-2022-91285772.html | ||
| test_url: https://www.bw24.de/baden-wuerttemberg/wolf-baden-wuerttemberg-sichtung-zollernalbkreis-spaziergaengerin-raubtier-nachweis-91297558.html | ||
| test_url: https://www.chiemgau24.de/chiemgau/chiemsee/gstadt-am-chiemsee-ort118608/gstadt-tourist-info-leiterin-berichtet-von-guter-auslastung-bei-gaestezahlen-trotz-corona-91288990.html | ||
| test_url: https://www.fnp.de/frankfurt/frankfurt-sachsenhausen-der-adlhochplatz-bekommt-einen-neuen-namen-91287869.html | ||
| test_url: https://www.fr.de/frankfurt/die-nfl-kommt-nach-frankfurt-91329620.html | ||
| test_url: https://www.hna.de/kassel/eigentuemer-sollen-sich-ruesten-91290102.html | ||
| test_url: https://www.kreis-anzeiger.de/lokales/wetteraukreis/mit-hoher-sicherheit-ein-wolf-91287739.html | ||
| test_url: https://www.kreiszeitung.de/lokales/bremen/bremen-wie-sich-osterholz-tenever-veraendert-hat-91304118.html | ||
| test_url: https://www.wetterauer-zeitung.de/wetterau/mahnwache-fuers-impfen-91318413.html | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like there can be a fingerprints from the html code.
That can be added over here: https://github.com/j0k3r/graby/blob/6ed01641205b7fbdea4ce9ba15a34989af9edc1a/src/Extractor/ContentExtractorConfig.php#L39
What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will check that tomorrow. Thank you for looking into that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this one is not working with fulltext-rss. As I wrote in #926 ftr seems to check only the
<head>for matches, but your suggestion is within the<body>. And to get sure, I also used a shorter version just for testing:'id-SiteBEEPWrap' => array('hostname'=>'fingerprint.ippen.media', 'head'=>true),But still got "No fingerprint matches"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@HolgerAusB We'll see if it's possible to add this to Full-Text RSS in the config file. It should be possible to use the body and not just the head, if not, we'll fix that in the next version.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for the slow response. Haven't forgotten about this, just been very busy. I think we can merge this, as the config changes if/where they're needed have to happen separately anyway. What do you think @j0k3r @HolgerAusB