How to put the results Yandex.Cards?

Good day!

Please tell me how to put data from your organization's page in Yandex.Maps? In particular, the need to take the data on the rating and number of reviews she has. I know that there is a API the link https://yandex.ru/maps/api/business/fetchReviews (found it among the issues on the same site), but it is not quite that. There are given the reviews themselves, and you need what I wrote above.

There is one problem — Yandex renders elements on the page gradually. When parse it using Simple HTML DOM, it is empty. How it can be avoided? Surely running a script that accesses the link and get all the data. Don't know how to handle it.
April 7th 20 at 10:55
3 answers
April 7th 20 at 10:57
Solution
Using Selenium to parse\Puppeteer
Thanks, I'll try them! - bobbie65 commented on April 7th 20 at 11:00
April 7th 20 at 10:59
Solution
That has the problem: to correctly extract data the rating organizations. Asked the support of Yandex, how to do it without breaking the rules of the platform. Waiting for a response and will post here.

In the meantime, there is a simple solution to extract data using PHP charCode:

<?php 

rating_function function($atts) {
extract(shortcode_atts(array(

 // insert the link to the card organizations in Yandex Spravochnike as is
 "url" => 'https://yandex.ru/maps/org/<---id organization--->',
), $atts));

 $content = file_get_contents($url);

 // define the beginning of necessary code snippet, which we will remove all content
 $pos = strpos($content, '<div class="orgpage-reviews-view__rating-score">');

 // delete all before the desired snippet
 $content = substr($content, $pos);

 // find the end of the necessary fragment of the code
 $pos = strpos($content, '</div>');

 // cut the needed number of characters from the end of the fragment
 $content = substr($content, 0, $pos);

 echo $content;

}

is the add_shortcode('rating', 'rating_function');

?>


Get a rating in a numeric value and inserted the shortcode where you want on the website: [rating].

PS: Only the function you need to run as rarely;) in order to avoid the load, preferably via a periodic refresh of the cache, for example, by closing the curl connection curl_close(); or each time causing the request form data manually.

Here is proposed one solution by means of curl.
This is not a php shortcode and WordPress shortcode - florida_Aufderh commented on April 7th 20 at 11:02
Almost what you need, thank you. I'll try to finished and get to work via curl. - bobbie65 commented on April 7th 20 at 11:05
@florida_Aufderh, Yes, you will need to use third party - Roxane commented on April 7th 20 at 11:08
@bobbie65, for one of my projects washed down by CURLe.


Post a working script to create the shortcode (WordPress) for the supplier.
Important: works at the library phpQuery
You can create a shortcode and PHP. There is a option BBCode.

<?php

/* Input data
* The organization address of the form: <a href="https://yandex.ru/maps/org/---id---">https://yandex.ru/maps/org/---id---</a>
* The Class in which is located the rating: 'orgpage-reviews-view__rating-score'
* To connect the library phpQuery: <a href="https://code.google.com/archive/p/phpquery/">https://code.google.com/archive/p/phpquery/</a>
*/

curlshot_func function() {

$get_url = 'https://yandex.ru/maps/org/---id---';
$refer = 'http://google.ru';

$headers = array (
 "Cache-control: no-cache",
 "Upgrade-insecure-requests: 1",
 DNT: 1,
 "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp",
 "Connection: keep-alive",
 "Upgrade-Insecure-Requests: 1",
 "Pragma: no-cache",
 "TE: Trailers",
 "Accept-language: ru-ru,ru;q=0.8,en-US;q=0.5,en;q=0.3"
);

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $get_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, THIS, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_ENCODING, 'gzip, deflate');
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
/*curl_setopt($ch, CURLOPT_PROXY, '127.0.0.1:8080');
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_HTTP);
curl_setopt($ch, CURLOPT_PROXYUSERPWD,'user:pass');*/
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_REFERER, $refer );
curl_setopt($ch, CURLOPT_COOKIEFILE, __DIR__ . '/cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, __DIR__ . '/cookie.txt');
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.154 Safari/537.36');
curl_setopt($ch, CURLOPT_VERBOSE,1); 

$data = curl_exec($ch);
if ($output === FALSE) {
 // If something went wrong
 echo 'Oops, a mistake:' . curl_error($ch);
return;
}
$document = phpQuery::newDocument($data);
$element = $document->find('.orgpage-reviews-view__rating-score')->text();
return $element; // don't echo, namely, the return correctly reflect the data in the shortcode
}
is the add_shortcode('curlshot', 'curlshot_func');


The script will save the shortcode value rating organizations: [curlshot] - Roxane commented on April 7th 20 at 11:11
April 7th 20 at 11:01
NO way.
YOU have lifted fans darmovschinki.
You reviews already I have to use it all the time shows a captcha, which hell read.
And Yandex, and so gives for free, through API. But no - we nose gate, we better be the change in his pockets tyrit, and shit all around.

Try at least once in life to make something of their own, and not try to steal someone else's.
And where does "rip-off"? Ya cards open allow to read all data of the organization, and the person just wants it all in Excel to drive. - Ozella.Kohl commented on April 7th 20 at 11:04
What a nonsense and idiotic opinions with level ground?

Like politely and humanly asked how to implement a certain function, which itself can do. I need a corny show on the website of the service his rating on Yandex and the link to the page with the reviews. What the fuck, "detail", "shit around" and "rip-off someone else"? What are you talking about?

I read the questions here on the portal — not found information about this. I read the documentation for the Yandex — again have not found how to do it. Came here and asked for help.

If you specifically life than I have offended, sorry. Instead pour all that you have accumulated would just say that at the moment this is not possible. - bobbie65 commented on April 7th 20 at 11:07
No need to build from a naive fool. You understand perfectly.
"I need corny in my home to eat potatoes that a neighbor had planted, watered, harvested, your labor invested. I politely asked her gently fuck on the table".

"How to display" is one question. "How to put contrary to the terms of service" - very different. - Dee_Ortiz98 commented on April 7th 20 at 11:10
@Dee_Ortiz98, here grow their potatoes at home alone, rather than to engage in mutually beneficial exchange with a neighbor. And others do not disturb ;) - bobbie65 commented on April 7th 20 at 11:13
Dada, have greedy children "mutually beneficial cooperation" means "I must". - Dee_Ortiz98 commented on April 7th 20 at 11:16
@Dee_Ortiz98, Monday - day heavy? - florida_Aufderh commented on April 7th 20 at 11:19
@florida_Aufderh, yeah, everyone who "liked" my answer.
It's not that one day is heavy, and that normal people understand that we live in a society and should not shit around. - Dee_Ortiz98 commented on April 7th 20 at 11:22
you need to take the data on the rating and number of reviews she
- and what's behind the theft?
Well, we have no centralized place where you can officially take this data here and collect them in the bazaars of the Internet.
This is not to put ads with Craigslist and on your portal to place. Here we need specific information from society - social organization from the people.

Don't understand what you under so burning a chair... - florida_Aufderh commented on April 7th 20 at 11:25
@florida_Aufderh, And I don't understand where is the theft. I picked potatoes in society. Goose. The car ride. Bye new. And whats the big deal?
You are the backend that collects data from the society was doing? For placing servers in the data center pay? No? Let the "society pays", and you nahalyavku use this?
Well, we have no centralized place

You or blunt or cringe
Yandex has API that can be used in a civilized way to recover the data. For free. As much as he is willing to share.
But that's not enough, you want to put from a web muzzle and to make Yandex more captchas to cover up. That is shit in the end all. - Dee_Ortiz98 commented on April 7th 20 at 11:28
@Dee_Ortiz98, Yandex captcha will hang without foreign encroachment. For the future.

And you see all the theft you see?

To attack me don't need your rudeness and leave parents. - florida_Aufderh commented on April 7th 20 at 11:31
@florida_Aufderh, see you whined "no attack on Minya dyadinka". I see theft where it is. Everyone must decide for himself how to dispose of their labor. I write articles and post them to share. And does not limit the viewing. But if someone will copy and use to promote your govnosayt - it will be a rip-off and theft. He can't do anything, so stealing the pockets of others. With the Yandex same. He makes use of twoism service, all for free. But to steal to use on your website - no. And this requirement must be respected. - Dee_Ortiz98 commented on April 7th 20 at 11:34
You there's something about potatoes, something about the iPhones inappropriately...

Just don't need to attack without understanding the purpose of obtaining data. Or you shop for knives, too hysteria - "and kill them! You in prison must be closed!" As it turns out.

It is strange to see your arguments about "I write"... Not write. No one will notice. And the fact that posted in the Internet already download who need. And incite your text AI. But that's if you study in your texts - will be wholly evil.

And about Yandex found someone to protect. The Corporation will gall you and your descendants.

Then found Greta of the Internet - florida_Aufderh commented on April 7th 20 at 11:37
@florida_Aufderhyou turn like a snake, if only to justify themselves for what they steal from their brother of the webmaster, even though deep down inside you realize that that's just an excuse. Well, move on. - Dee_Ortiz98 commented on April 7th 20 at 11:40
@Dee_Ortiz98, a full moon only last night... That I was stealing? - florida_Aufderh commented on April 7th 20 at 11:43
@Dee_Ortiz98, Your concerns and wishes are fully justified. Stealing is not good. Let's face it, that is theft and who is more likely the rights of the Customer or the Service.

The Yandex Directory as the same as the whole IT industry is developing gradually and the possibility to extract information about the APi they have, to date, no!


We, as they say one thing to do:) and the whole Internet is sharpened by the target user needs and if the needs of the site, the company needs its own data from public sources - it is not theft and theft. Moreover, under the new policy GDPR user of any service may request the export of your data or to require their deletion. Can present if the Service will not hear him.

Today, customers need to structureat their data from different sources: Google, Yandex, tripadvisor, social networks and so on. No one is going to hack and load all the system to parse data from closed sections, to penetrate, to rewrite their code from different browsers using JS. And there's no crime that the company wants to get its data from public sources.

Rights to their data belongs to the Company, not the service. Services can then de jure be punished and fined for what they removed without permission, location data, address, brand and so on. And I think such violence (who owns the data) even touches the Internet in the coming years, when the GDPR will be introduced and we have in Russia.

The company is not guilty of that services reviews are not developed for the needs of each individual APi. But at the same time, information about companies, all operated by paid and free.


In this case, the developers decide the needs of the customers way and of course, all the rules of ecology programming. The main rule of "do no harm" works not only in medicine.

I made a request in Yandex on obtaining a rating of the Companies for their urban projects. Yandex confirmed that such APi yet, but my wish sent to the Department of development and may soon be. Waiting for updates.

5e4278762aab3862191780.jpeg - Roxane commented on April 7th 20 at 11:46
@Roxane, thanks for such details explanation. Honestly, not even thought about the fact that all the data in Yandex.Directory and search services legally do not belong and as it is taken from other open sources. Show in their results the same original photographs from the sites of the companies, for example.

And indeed, they exploit them as they please, and earn well on this. Even banal — people looking for information on the company, and he then shows targeted advertising and other services from the same sector. A good scheme of earnings obtained. So that the line on the legality of certain actions by private programmers and large companies, is very thin.

It is strange that to information to bypass the API, they are not allowed, although we really do not harm anyone. In General, wait until officially drop this opportunity. And yet — try to do it manually. - bobbie65 commented on April 7th 20 at 11:49
@bobbie65, the word "parse" offends the feelings of the developers:)) and sounds like sedition against the user agreement Yandex. Come on, it's the word leave for hackers darknet and the Union will use such phrases as: to extract data from open sources:)

In fact and in truth, will present the situation as freelancers go to dozens of services and thousands of companies that handles verify and update the data... it's better in the Cabinet of udalenie just press the button to request information and to apply.

And this, I believe, is not theft and crime, because the data is retrieved from open sources to publish in open access with an active link to the source. It's like the frame to open the video from YOU tube. Don't see anything wrong with that, on the contrary, solid pros and Yandex additional traffic + promotion platform in the face of the owners of the cities. - Roxane commented on April 7th 20 at 11:52
@bobbie65, the U.S. court of fully legalized scraping sites and banned him technically to prevent!

https://habr.com/ru/company/globalsign/blog/466911/

LinkedIn technical measures to block the scraping prevent hiQ contracts with private customers who rely on these data. In legal jargon this is called "malicious interference with contract" (tortious interference with contract).


In both! If a website takes measures to protect from scraping open data is malicious interference with contract. So legally, scraping open data the sensible thing. And in Russia similar lawsuits yet to come.

The only thing that you need to parse like a browser.

The court not only legalized this practice, but was forbidden to interfere with the competitors to remove the information from your website automatically, if site is public. The court confirmed clear logic that entering the bot-scraper legally no different from entering the browser. In both cases the user asks to open data and doing something with them on my side.

Now, many site owners are trying to put technical obstacles to competitors who copy their information is not protected by copyright. For example, the prices for tickets, merchandise lots, open user profiles, etc.

Some sites consider this info "proprietary" and scraping regarded as "theft". Legally it is not so that it is now officially fixed in the United States.
- Roxane commented on April 7th 20 at 11:55
@Roxane, it is reasonable.
Even better, in General - Myra.Roberts78 commented on April 7th 20 at 11:58
@Roxane, very good news. We will know) - bobbie65 commented on April 7th 20 at 12:01
it remains to understand what does the decision of the court amerikanskogo to the user agreement for Yandex. - Dee_Ortiz98 commented on April 7th 20 at 12:04
@Dee_Ortiz98, well, the court decision forms a trend, but Yandex is quite affordable, it seems to be from the US too. So that is indirectly related. - Myra.Roberts78 commented on April 7th 20 at 12:07
@Dee_Ortiz98, judicial precedent in the United States will serve to the triumph of common sense, when the developers and their customers will be able to sue for lost profits caused by the imperfections of the current legislation. No law, no agreement and the contract cannot cause harm and create obstacles for the development of society and healthy competition. - Roxane commented on April 7th 20 at 12:10
@Roxane, Dada, subtract and divide.
One provides the infrastructure, salaries, support, actualization of data. And the second wants it all for free. "Healthy competition", yeah. - Dee_Ortiz98 commented on April 7th 20 at 12:13
@Dee_Ortiz98, You misunderstood. In the context of not talking about free or stolen content. Just scraping in a safe manner not to the detriment of the resource and only open data. For example, this topic is dedicated to extracting data from the reviews. This may be the rating company.

Go to Google search write to any famous place and You will see the card Google not only your rankings and reviews, but the rating of Yandex + of Ottavio.

How do You think Google violated user agreement for Yandex, which itself is a lovely soul parses a Google. This allows users to have objective data about the institution and promotes healthy competition among institutions. - Roxane commented on April 7th 20 at 12:16

Find more questions by tags PHPParsing