[ERROR] CRITICAL:twint.run:Twint:Feed:noDataExpecting ~ Inconsistent results [High Severity]

[ghost]

Python 3.6
twint 2.1.7 updated from master
Have searched issues without finding anything
Running on Ubuntu 18.04, anaconda, jupyter notebook

Commands run:

import twint
import pandas as pd

c = twint.Config()
c.Pandas=True
c.Search = "#nfl"
c.Hide_output=True
c.Since = '2019-12-01'
c.Until = '2019-12-02'

twint.run.Search(c)
df = twint.storage.panda.Tweets_df

Hi, thanks for writing this package, it's very useful. I'm clearly not using it right though. I ran the commands above as a test, using"#nfl" as a query because it's innocuous and guaranteed to have a lot of results over the course of one day, but I am getting inconsistent results.

First, when I run it I get a lot of these warnings (which I saw from another issue are probably related to http/https?): CRITICAL:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
That's fine though, the script still runs. The problem is the results are inconsistent. I ran it last night and got back 6,832 tweets, then ran it again this morning as part of testing some other code and got 4,710 tweets. When I saw that I ran it again and got 0 tweets.

I have a couple of questions if that's okay. Is twint caching the results of queries somewhere, and if so, how do I clear the cache? Is this inconsistent behaviour expected (is it a Twitter search page thing?) and if so, does it make sense to run the same search multiple times and concatenate results? Finally, is there a suggested best practice for searching date ranges? (i.e. If you want all the tweets for a hashtag for the past 3 months, is it better to do one big search or break the search into daily or weekly time ranges?)

Again, thanks for this package. Great work.

[pielco11]

Twint does not cache results, queries or anything else. Every single piece of data is provided by Twitter. It makes sense to run multiple searches until it makes sense to you, for example you could monitor specific hashtags and see which user deletes more tweets than others. On the other side, you will most probably get duplicates if you don't filter the data with since and until (or other parameters).

If the hashtag is really popular, I'd split the searches in months if not even weeks.

[ghost]

Thanks. So, is it normal though that if I run the exact same search for the same query 12 hours apart that I get very different numbers of tweets back? When I ran it yesterday I got ~6800 results and when I ran it again this morning I only got ~4700. (When I ran it a third time it returned 0 tweets??)

It's not really that big a deal, I'm just hoping to understand the expected behaviour to explain to analysts when I explain the data to them. If I'm only getting a subset, or a random subset of the data, that's fine, but I need to understand that before they ask me about it.

Thanks

[pielco11]

If you run the same script over time, so with the same date-time ranges and other parameters, the dataset should be "constant" since it does not depend by the relative time which you run the script at.

So if we have a variation and what can variate is the time interval or the dataset itself, I guess that (in this case) is the dataset that's changing. Most probably due to deleted tweets, accounts that go from open to closed, accounts that get delete/suspended.

Let's say that tweets with id = 1,2,3 are sent in the interval time (A,B). If we run the scripts at C with C > B, and the tweets don't get deleted/shadowed et similia , we'll always get tweets with id = 1,2,3. The interval is closed and nothing can go it or get out by itself. The only way which a tweet can go out of that interval is just being deleted/shadowed and so on.

[ghost]

Thank you. I guess my strange results were a glitch. I mean, it's unlikely that 2000 tweets got deleted for that hashtag overnight. I'll keep experimenting.

Much appreciated.

[pielco11]

I can guarantee you that what Twint returns is what Twitter gives (and this can be proven, just run with --debug or config.Debug = True and in the twint-request_urls.log file you'll see every request made, and you can replicates the request with any software of your choice).

You can run the script with your handle as target, if you always get the same tweets everything is fine otherwise it needs to be investigated.

Best of luck!

[ghost]

Thanks, I'm sure it's something I'm doing. I will keep at it.

[ghost]

Sorry to come back to this, but just wanted you to be aware of the results of my testing.
I ran this exact code multiple times in a row, and each time it returned very different results. I don't know if there is something wrong with my code (please tell me if so!) or if there is something else going on.

(running with python 3.6 and fully updated twint)

c = twint.Config()
c.Hide_output=True
c.Pandas_clean=True
c.Pandas=True
c.Search="#nfl"
c.Since='2019-12-01'
c.Until='2019-12-02'

twint.run.Search(c)

I just ran that exact code 4 times in a row. It returned this many tweets:
Run #1: 1,909 tweets
Run #2: 280 tweets
Run #3: 13,207 tweets
Run #4: 3,015 tweets

I'm really not sure what to do with that at this point? Am I doing something wrong? On the three runs with the lower values, it also returned this

CRITICAL:root:twint.run:Twint:Feed:Tweets_known_error:Expecting value: line 1 column 1 (char 0)
Expecting value: line 1 column 1 (char 0) [x] run.Feed
[!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it!

Any advice would be welcome.

Edit: For what it's worth, it's definitely something to do with the feed being disrupted. That error is being thrown in the Feed method of Twint, and it seems to happen most of the time. I re-ran that script in a loop a bunch of times, and 13, 207 tweets seems to be the actual correct number, but it doesn't come back with that very often.

[pielco11]

I've tried your query and I can confirm that the results are not consistent. That's really strange and needs to be investigated.

I'll keep you updated

[ghost]

Thanks!

[pielco11]

So it seems that having HTTPS or not does not always have effects, for now my findings are there (I've run the script 3 times):

==> nfl1.csv <==
1201274724778565632,1201274724778565632,1575241187000,2019-12-01,23:59:47,CET,1180200593962369024,yotesglendale,GlendaleCardinals,,Brandin Cooks leaves Budda Baker in the dust💨 #NFL #NFLSunday #LARams #LAvsAZ #redsea pic.twitter.com/QowdS89R0S,[],[],[],0,0,0,"['#nfl', '#nflsunday', '#larams', '#lavsaz', '#redsea']",[],https://twitter.com/YotesGlendale/status/1201274724778565632,False,,1,,,,,,,"[{'user_id': '1180200593962369024', 'username': 'YotesGlendale'}]",,,,

==> nfl2.csv <==
1201274724778565632,1201274724778565632,1575241187000,2019-12-01,23:59:47,CET,1180200593962369024,yotesglendale,GlendaleCardinals,,Brandin Cooks leaves Budda Baker in the dust💨 #NFL #NFLSunday #LARams #LAvsAZ #redsea pic.twitter.com/QowdS89R0S,[],[],[],0,0,0,"['#nfl', '#nflsunday', '#larams', '#lavsaz', '#redsea']",[],https://twitter.com/YotesGlendale/status/1201274724778565632,False,,1,,,,,,,"[{'user_id': '1180200593962369024', 'username': 'YotesGlendale'}]",,,,

==> nfl3.csv <==
1201274724778565632,1201274724778565632,1575241187000,2019-12-01,23:59:47,CET,1180200593962369024,yotesglendale,GlendaleCardinals,,Brandin Cooks leaves Budda Baker in the dust💨 #NFL #NFLSunday #LARams #LAvsAZ #redsea pic.twitter.com/QowdS89R0S,[],[],[],0,0,0,"['#nfl', '#nflsunday', '#larams', '#lavsaz', '#redsea']",[],https://twitter.com/YotesGlendale/status/1201274724778565632,False,,1,,,,,,,"[{'user_id': '1180200593962369024', 'username': 'YotesGlendale'}]",,,,

So as we can see, Twint starts always at the same point. Which is good.

Now we have to see where it stops

==> nfl1.csv <==
1201248296645398528,1201248296645398528,1575234886000,2019-12-01,22:14:46,CET,178163508,randi_heatlifer,Randi Hilsercop,,The jets are so bad that the bengals just beat them 😂😂 #NYJvsCIN #NFL  pic.twitter.com/5OEHqX1I45,[],[],[],0,0,0,"['#nyjvscin', '#nfl']",[],https://twitter.com/Randi_heatlifer/status/1201248296645398528,False,,1,,,,,,,"[{'user_id': '178163508', 'username': 'Randi_heatlifer'}]",,,,

==> nfl2.csv <==
1200912390356783104,1200912390356783104,1575154800000,2019-12-01,00:00:00,CET,4059670933,blowoutbuzz,BlowoutBuzz,,YourDozen: Get your NFL Week 13 picks in to win 2017-19 sets >>  http://bit.ly/2Ok7pdQ  #collect @PaniniAmerica #TheHobby #NFL #predictions #picks pic.twitter.com/ZQFTduKvrh,['paniniamerica'],['http://bit.ly/2Ok7pdQ'],['https://pbs.twimg.com/media/EKkj3X2WsAALfcr.jpg'],0,0,0,"['#collect', '#thehobby', '#nfl', '#predictions', '#picks']",[],https://twitter.com/BlowoutBuzz/status/1200912390356783104,False,,0,,,,,,,"[{'user_id': '4059670933', 'username': 'BlowoutBuzz'}, {'user_id': '44128979', 'username': 'PaniniAmerica'}]",,,,

==> nfl3.csv <==
1201245479746646022,1201245479746646022,1575234215000,2019-12-01,22:03:35,CET,2691199254,seahawksreddit,/r/Seahawks,, https://ift.tt/2DyJ51V  Ravens Win! #Seahawks #NFL #GoHawks,[],['https://ift.tt/2DyJ51V'],[],0,1,6,"['#seahawks', '#nfl', '#gohawks']",[],https://twitter.com/SeahawksReddit/status/1201245479746646022,False,,0,,,,,,,"[{'user_id': '2691199254', 'username': 'SeahawksReddit'}]",,,,

If we take a look at the twint-last-request.log file (when Twint exists with error)

      <div id="main_content">
            <div class="system">
      <div class="blue">
        <table class="content">
          <tr>
            <td>
              <div class="title">Sorry, that page doesn't exist</div>
              <div class="subtitle"><a href="/">Back to home</a></div>
            </td>
          </tr>
        </table>
      </div

If we take a look at the latest scraped tweet in nfl2.csv we see that its time is 00:00:00 (relative of my TZ) which is good, it says that we reached the end of "my day".

A note about the time-zone. If we run the same with two different local times, most probably we'll get different results since my start (end) of the day is different that yours. That said, our aim is not to be sure that one gets each other's results, our aim is to get the same results each time we ask for them, comparing results individually. (FYI I got 25282 tweets)

Reasons why the issue might be related to HTTP(s) switch:

• since the switch we've had error messages
• in the past we've tested HTTPS and got the error messages

Reasons why the issue might not be related to that switch:

• the error messages are printed at first, then those vanish
• for valid requests of every run (HTTP or not), the content (tweets) is always returned

What happens when Twint gets those error messages:

1. Twint changes the UserAgent
   

   twint/twint/run.py

   Lines 89 to 94 in 3a4f778

   	logme.critical(__name__+':Twint:Feed:noData' + str(e))
   	# Sometimes Twitter says there is no data. But it's a lie.
   	consecutive_errors_count += 1
   	if consecutive_errors_count < self.config.Retries_count:
   	self.user_agent = await get.RandomUserAgent()
   	continue

2. Twint re-runs the same request (since self.feed and self.init don't change, the request's params are still the same)

Sometimes, luckily, error messages are not printed, even if running the same query. In such case the only thing that Twint has different than in the other runs is the UserAgent.

So maybe Twitter plays differently based on the UserAgent specified.

Updates soon.

[pielco11]

It seems that using Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2224.3 Safari/537.36 as UserAgent (as suggested by @o7n in #587), allows us to get almost all the expected results

So I suggest you to edit lines 158 and 159 in url.py and to specify that user-agent, or another one suggested. Then please run and let me know if you get consistent results @jomorrcode

[ghost]

Hmmm. My installed url.py only has 146 lines (installed twint 2.1.9 with pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint as per the instructions).

I looked through it and couldn't find any reference to user agent. I found it several places in run.py, but I didn't want to just start randomly changing the code without knowing that I was doing.

[pielco11]

Sorry I mean in get.py, my bad

twint/twint/get.py

Lines 155 to 161 in 3a4f778

	async def RandomUserAgent():
	logme.debug(__name__+':RandomUserAgent')
	try:
	ua = UserAgent(verify_ssl=False, use_cache_server=False)
	return ua.random
	except:
	return random.choice(user_agent_list)

Here is you'll end up with

[ghost]

Sorry, finally had a chance to try this. Yes, I made that change with that user agent and ran the code 4 times, and got back 13,011 tweets each time.

[ghost]

Just to update, running with that user agent ususally seems to bring back a fairly complete list of tweets, but does randomly fail and a smaller subset are returned. I am playing around with a workaround to look at the last tweet returned to see if it's timestamp is close to 00:00:00, and if not, redo the query. Not sure if there's a more effective way to detect that the scrape finished early as a workaround.

[o7n]

It would help a lot if there was some way to know if there were any errors during the search or profile request. Now the only indication of something going badly is a message on stderr, if any.

[ghost]

I thought I'd just add that after a lot more experimenting, results continue to be inconsistent regardless of the User Agent, though quite randomly so. Sometimes I can run the same code 3 times and get the exact same number of tweets and other times it will return a much smaller number, or even zero tweets.

[pielco11]

And does in happen regardless the query?

[ghost]

Huh, oddly it does seem to vary with the query. Some queries I tried seem to always return the same number, others vary a lot. Really strange that. That football hashtag (#nfl) is always very variable, but something like #france or #germany seems to be consistent.

[pielco11]

That makes the debugging even harder, as of now I'd exclude a flaw in Twint. So I guess that somehow the issue is related to Twitter

It'd be interesting to check if it returns less tweets even if it reaches the end of the day. Because if it stops before for unknown reasons, you could just resume from that point.

To try this out just run something like twint -s "#nfl" --debug --resume "test_1.session" --since "2019-12-18" --until "2019-12-19" --csv -o "test_1.csv"

import twint

c = twint.Config()
c.Search = "#nfl"
c.Debug = True
c.Resume = "test_1.session"
c.Since = "2019-12-18"
c.Until = "2019-12-19"
c.Store_csv = True
c.Output = "test_1.csv"

twint.run.Search(c)

If it does not stop at 00:00:00, you can just re-run the script/command as is and it will resume from where it started. You might want to apply some more complex logging, but keeping track of session files would be enough. Please consider that at every run the session file is overwritten, so you would like to do something like python3 script.py && cat test_1.session >> history.sessions or twint .... >> history.session. So that we can compare the session ids with the debug files (twint-request_urls.log is not overwritten at every run)

[ghost]

Oh, that is perfect. I will try that. I was trying to do something like that myself by checking for whether the last tweet collected was close to 00:00:00, but when it wasn't I had to rerun the whole script instead of just restarting from where it stopped.

So if I understand correctly, as long as the script hasn't terminated, re-running twint.run.search(c) will restart from the last tweet collected (assuming it hadn't reached 00:00:00), so a simple loop with a check for the latest time collected should do the trick.

Thanks, you've been very helpful with this.

[0m3rta13]

Going to echo what others have said that this probably isn't anything to do with twint.
Not sure if anyone else who uses the Twitter Dev Api received the string of mails recently that myself and a few people I know have?
The initial one of concern spoke about Dev Apis possibly being abused after being exposed through the browsers cache etc etc.

Seems to me that coupled with the seemingly "new" policies Twitter have been enforcing recently like forcing number verification, banning random accounts and all that I am of the opinion that they are simply starting to limit the type of full access that we've all become used to & just a general crackdown of sorts

I've been having luck with a solution @yunusemrecatalcam posted in issue #888 https://github.com/twintproject/twint/issues/888#issuecomment-693977671 So big thank you to him and the Dev of the project mentioned. Works nicely with a rotating pool, speeds are not that bad at all

And of course Big thank you to @pielco11 and everyone involved for your awesome work on Twint. It really is appreciated!

[hshafiq132]

I have also faced the same issue. Twint is not working anymore and giving the error as given in question. How to get rid from this. Is there any alternate ?

[JiaoZiLang]

looks like it started again -_-

[LinqLover]

Update for all interested people: Looks like this issue has already been resolved an unknown time ago, if you clone the current version of twint, the noDatExcepting error will not occur any longer. (via #983 (comment))

🎉

[LinqLover]

When will these changes be released on pip?

[aliabdmahdi]

I need to collect tweets, so I used the same code, but I have a problem with implementation .. Please help me because I am a student and the time is very short to complete the project.
import twint
import nest_asyncio
nest_asyncio.apply()
c = twint.Config()
c.Hide_output=True
c.Pandas_clean=True
c.Pandas=True
c.Search="#covid"
c.Since='2020-12-01'
c.Until='2020-12-10'
twint.run.Search(c)

[himanshudabas]

@aliabdmahdi
Instead of installing twint from pypi, use the below command:

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

[aliabdmahdi]

Uninstalling the Twint tool, then I used this code (pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint). This error appeared to me. I don’t know the reason ?? .

[himanshudabas]

It's because this method first clones the twint from GitHub, that requires you to have git installed on your system.
Instead what you can do is, download the twint repo from GitHub, extract it. And then open the anaconda terminal in the extracted folder.
Then run the following command:

pip install .

[aliabdmahdi]

It's because this method first clones the twint from GitHub, that requires you to have git installed on your system.
Instead what you can do is, download the twint repo from GitHub, extract it. And then open the anaconda terminal in the extracted folder.
Then run the following command:

pip install .

How open the anaconda terminal in the extracted folder? .

[himanshudabas]

First start the conda terminal, like you previously did. Then change the directory using the cd command.

Google how to change directory in cmd

[aliabdmahdi]

I put the extracted tool file on my desktop and then went through the installation steps , and there was also an error. What is the problem now?

[aliabdmahdi]

I put the extracted tool file on my desktop and then went through the installation steps , and there was also an error. What is the problem now?

[0m3rta13]

I put the extracted tool file on my desktop and then went through the installation steps , and there was also an error. What is the problem now?

As per the the error. If you read it it asks you a question.

Do you have got installed?
It's asking because you don't have it installed. Or it's not in your PATH. But I'm gonna guess it's not installed.

Have you installed git?

If not....

https://git-scm.com/downloads

[aliabdmahdi]

after install git , After performing the steps, this error appeared ..

[0m3rta13]

after install git , After performing the steps, this error appeared ..

Don't take this the wrong way but I think you should really consider getting to know the things behind the tools you're trying to use before using them.
For example, in this case, python.

That error indicates that you're probably not using a correct or compatible version of python. Or you've only just done the initial install of conda and not updated it prior to using it.

A quick google search of the error you're getting would help you more than waiting for replies here. And you'll probably learn a a bit along the way which will only be good.

Again, I'm not trying to be mean. But you're going to find yourself running into many issues like this when you start out and people aren't going to want to help you if you don't appear to really try and fix your problem yourself first. Like by just googling your error.
Also because the problem you're having is in no way related to Twint or it's Devs and this space is really for supporting and helping people who experience problems with their tools. It's good practice to respect that.

That error points to countless posts on stackexchange for example. And they always offer up great, detailed information that'll really help you.
I used stackexchange for years before I even posted my first question because almost anything I've ever wanted to ask has already been asked, many times. Lol.

Just some advice. Hope it lands and comes across as intended.

[aurigandrea]

I'm afraid, the "CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1" error is back. uninstalling + pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint doesn't work now. Am I the only one for whom this issue is back?

[Lukpier]

I'm afraid, the "CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1" error is back. uninstalling + pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint doesn't work now. Am I the only one for whom this issue is back?

same here ..

[aravind-naidu]

I'm afraid, the "CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1" error is back. uninstalling + pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint doesn't work now. Am I the only one for whom this issue is back?

I am getting this same error!

[MoHayat]

I'm afraid, the "CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1" error is back. uninstalling + pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint doesn't work now. Am I the only one for whom this issue is back?

same here, uninstalled it to try this trick and ran into this issue while reinstalling.

[MoHayat]

UPDATE: for MacOS 11.0 & Python 3.7

• pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint keeps failing because it can't build the wheel for pycares. If you have homebrew installed, running brew unlink c-ares fixes this issue (according to this SO answer)
• after successfully installing twint, I noticed that all of the issues causing this error for my use case (twint.run.Search(query), specifically), were type errors, so I forked the project and fixed them (you can check out the branch on my fork here)

everything works like a charm on my end now!

I'd be more than happy to submit a PR, but decided to hold off until I can get some confirmation that this is a valid solution, or whether other use cases bring up more of these errors.

[aurigandrea]

UPDATE: for MacOS 11.0 & Python 3.7

• pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint keeps failing because it can't build the wheel for pycares. If you have homebrew installed, running brew unlink c-ares fixes this issue (according to this SO answer)
• after successfully installing twint, I noticed that all of the issues causing this error for my use case (twint.run.Search(query), specifically), were type errors, so I forked the project and fixed them (you can check out the branch on my fork here)

everything works like a charm on my end now!

I'd be more than happy to submit a PR, but decided to hold off until I can get some confirmation that this is a valid solution, or whether other use cases bring up more of these errors.

Thanks! I got this error when I try your solution: "Error: No such keg: /usr/local/Cellar/c-ares"

[MoHayat]

@aurigandrea interesting, what error message do you get when you run the pip install command?

[LinqLover]

FWIW, we are running twint on an hourly basis (https://github.com/Museum-Barberini-gGmbH/Barberini-Analytics) and haven't seen any major issues during the latest months. We install twint using the pip dependency git+https://github.com/twintproject/twint.git#egg=twint.

[eledroos]

Is anyone else still having this issue? I keep getting Expecting value: line 1 column 1 (char 0) [x] run.Feed[!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it! on Windows with Python 3.8.8 twint 2.1.20 on every search I run. :(

[charliehawco]

Is anyone else still having this issue? I keep getting Expecting value: line 1 column 1 (char 0) [x] run.Feed[!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it! on Windows with Python 3.8.8 twint 2.1.20 on every search I run. :(

same here, what am I doing wrong?

I'm getting error messages like

CRITICAL:root:twint.get:User:'NoneType' object is not subscriptable
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)

[eledroos]

@charliehawco a fork has been working for me as of this AM -- not sure what's been going on with the core repo though.

Uninstall twint and reinstall with pip install git+https://github.com/woluxwolu/twint.git

See here for more details.

[charliehawco]

Hi @eledroos, your fork works brilliantly! only thing is the changes make to the format.py file don't appear to have make to the fork - see issue #1232