Are these bots good or bad?

[Randicus Draco Albus]

Being completely unsavvy about the internet, I have a very ignorant question.
I am tremendously curious about bots. One of the active members that is almost always present at DFN is Bing [Bot]. And today I noticed on DUF:

Registered users: Google [Bot], Randicus Draco Albus, Yandex [Bot]

I can understand the presence of Google's bot. It will be gathering information for its search engine results. But what are Bing and Yandex bots? The only thing I know is that bots have many purposes; some helpful, some benign and some harmful. Are these innocuous, spammers, government spy agencies or general purveyors of mischief? Should I be worried, ambivalent, unconcerned or happy? The more I know about such things, the more potentially useful I can be to any forum I am a member of.

I warned you it would be an ignorant question.

[dilberts_left_nut]

Just search bots indexing everything they can find ...

https://duckduckgo.com/?q=Yandex

[Randicus Draco Albus]

So there is no need to worry about the bots I can see. Thanks. Although, Bing Bot should be banned, because it is Microsoft.

[cynwulf]

Although, Bing Bot should be banned, because it is Microsoft.

If mickeyshaft want to "waste" their resources indexing GNU/Linux sites, then I say let them get on with it...

The yandex bot is a russian search engine if I recall correctly - it wasn't in the "bot list" when the forum was first set up, but it was showing up here quite a lot so I decided to add it.

[Randicus Draco Albus]

Thank you Left Nut very much for that link! Normally I cannot access DuckDuckGo in China, but I can not only access that link, but I was able to use it to reach DDG's home page????? Bizarre. Life is strange, in deed. Asking a stupid question brought DuckDuckGo back to me.
You get a fist-full of beans!

[nadir]

did you try the duckduck-onion ?
http://3g2upl4pq6kufc4m.onion/

can't say i know much about what get's censored where, why, and by whom exactly... might be i am telling the obious.

[Randicus Draco Albus]

No. And I cannot look at that link, surprise, surprise. The weirdest things are censored in China. For example, I cannot access images posted on Image Shack. My guess is that the internet monitoring software blocks anything with certain key words and/or uses certain servers or some such. A hypothetical example of what I mean would be if sites containing the word revolution were blocked, one would not be able to access something about the Industrial Revolution.

I have Tor installed, but do not use it, because it slows things down too much. If I was worried about the government spying on my activities, I would move to Taiwan. But having half the internet cut off, because of the government's paranoia is irritating.

[dilberts_left_nut]

Yes when I was in China the blocking seemed a bit random to me.
My work domain name (among others) wouldn't resolve, but I could access it by IP.
I didn't have the time to investigate too much so I proxied everything through my work server via an ssh tunnel.

What DNS servers do you use?
Can you use, say, googles public dns at 8.8.8.8? Does it give different results?

[Randicus Draco Albus]

At the moment I am living on a university campus that has its own server, so I use a static IP address and DNS. If one uses Google, all of their traffic now goes through Hong Kong, but is still controlled by Beijing. Wonderfully complicated.

The situation is complicated by connection speed. There are to few internet lines, so accessing anything is often difficult, due to too much traffic. Sometimes it is difficult to know if a connection cannot be established, because it is blocked or because the line has reached its capacity limit.

Can you use, say, googles public dns at 8.8.8.8? Does it give different results?

Never tried such things. Perhaps it is a worthy future experiment.


I just received a PM at DFN with useful information. I was informed that, because of all the spam that originates in China, the spam-blocking software used by some sites occasionally results in collateral damage (people like me). So some of my difficulty may be web-sites mistaking me for a spam bot.
At least I hope it is a mistake.