HackerOne users: Testing against this community violates our program's Terms of Service and will result in your bounty being denied.

How would you build a form whose back-end data is based on scraping?

I have a site, call it www.xyz.com, and it has a search box that you search for various things. For example, vanilla forums has a search box and when you search something, discussions pop-up in a structured format, and that's how my xyz site is.

I am using Python to automatically search a word on xyz, and then pull the content of the first "discussion" (Python can extract contents based on html tags)...Now where I am REALLY struggling is figuring out how I can make a PHP form on Vanilla that asks the user for a keyword, and then it parses that keyword to my Python app which outputs the extracted contents in response.

My initial solution is to get the python app to store the extracted contents in a MySQL db, and then get php on vanilla to output the MySQL contents for that keyword if that keyword exists in the previously queried list of keywords...If a user searches a keyword and it has not been previously queried, and is not in the previously queried list, then we have a DB table that lists "keywords to query queue", so then Python app checks at an interval to see if there are items in the "keywords to query queue" and then it querys them, remove them from that table, and stores the data extract inside the previously queried list.

That's my thought, but could someone suggest a different/better way? Anyone have experience with this?

Sign In or Register to comment.