Import of Endangered SE-Questions

As announced in my last post, I like to introduce the prototype of our new Q2A-plugin for the import of endangered SE-questions. For the user it has become the simplest and most comfortable solution I can imagine. Starting point is the link to the question on any SE-site loaded in your browser, as for example:

Copy this link. Note that the complete link is required; do not use the shared links at the bottom of the questions. Then you may select the menu option “Import SE-Question” on our Physics Overflow site, which is only visible and accessible to dedicated users like administrators or moderators (selectable by the super administrator):

Paste the link copied from the SE-site into the appropriate field of the import dialog:

Select the desired Physics Overflow category and click the import button. In a little while, the process announces the successful import of the complete thread containing the question and  all answers and comments:

The import is made using the StackExchange API. This API implements throttles, which reduce the number of daily calls to 300 for a single IP, as long as the application has no valid access token. If the application has an access token (obtained via authenticating a user), this number is 10’000 calls per day and per IP. My plugin requires typically two calls for each import (one for the thread and a second for the user data), as long as no more than 30 users have contributed to the question. For every 30 users more, again a call is required (I have found questions with more than 100 contributing users). This means that without an access token, about 150 questions per day may be imported. I have no idea what happens, when this quota is trespassed. The API returns the remaining quota of calls, which is divided by two in our plugin and indicated in the dialog window (see image above). A part of an example import is shown in the next picture:

Attribution

Attribution is regularized in the API terms of use, which point to the Stack Exchange Terms of Service. As far as I understand, we are allowed to copy content from SE-sites, as far as we follow the rules under this last link. My proposition is to put an attribution line under every imported question, answer and comment, that looks like this:

Like this, the SE rules and the rules of the  Creative Commons Attribution Share Alike license should in my opinion be fulfilled. The exact date and time of the import is added, because it is not possible to synchronize edits that are made on SE after the import. So the import is a snapshot of the state at the time indicated by this date/time. The API also provides no way to import the edit history of the questions.

If anybody has more knowledge about attribution to SE, I would be glad to get some feedback. By the way, shouldn’t we also think about terms of use for our site?

Remaining Issues

There are some issues on importing user identities, which I try to explain below. Users are imported exactly the same way as during the migration of the closed SE.TP, with their display name and email hash. The following cases may occur:

• User no more registered on SE-site. In this case, there exists no link to the user profile on the SE-site. The plugin then allocates the post to a user “UnknownToSE”, which is hidden in the list of users, similar to the voter introduced for the import of SE.TP questions.
• Collision with an existing user name on Physics Overflow. A user has registered with the same display name on PO as the user to be imported. In this case, the plugin checks the email hashes of both users. In case of a match, the imported user is assigned to the existing user. If the hashes are different, I have not yet a useful solution. Actually, I use again the user “UnknownToSE”, but this is not a good solution. Any ideas?
• Collision between identical users from different SE-sites. A StackExchange user may post for instance on SE Physics and also on SE Math, but using different email addresses. I have observed that such cases appear quite often. In contrast to user IDs on different sites, the only stable ID is the account ID of a user. Using the StackExchange API, it is possible to find this ID for active SE-users. However, the Area 51 dump did not provide this ID.

Any ideas for the solution of these issues are helpful.

Next Steps

I think it is slowly time to prepare the takeoff of Physics Overflow. In my next post I will make a proposition for this process. I hope Dilaton will have recovered soon and will be on board again. Get well soon!

Advertisement