Skip to:
Content

Opened 3 years ago

Closed 2 years ago

#1592 closed defect (fixed)

bbPress Standalone Importer breaks if source forum is too big

Reported by: ali3n0 Owned by:
Milestone: 2.1 Priority: normal
Severity: normal Version:
Component: Importers Keywords: dev-feedback
Cc:

Description

Seems there's a memory leak somewhere, given that even growing allowed memory indefinitely it still breaks pretty much at the same point. You can see the error in the attachment

Attachments (1)

bbpress20_migrate_fatal_error.jpg (230.7 KB) - added by ali3n0 3 years ago.

Download all attachments as: .zip

Change History (8)

comment:2 cnorris233 years ago

  • Keywords reporter-feedback added

Is anyone with this problem willing/able to provide some database files to test this out? It would take some work to produce forum data of that magnitude manually. Alternatively, does anyone know of a plugin that one could use to create database info, similar to http://wordpress.org/extend/plugins/demo-data-creator/?

comment:3 anointed3 years ago

@cnorris23

I don't have a bbpress forum, but can provide a vbulletin forum db.
Threads: 60,982, Posts: 295,135, Members: 26,800

We also have access to forums with 10million+ posts and 500k members if that would help.

*I've currently got one of the vb devs working on building me an importer for bbpress plugin. He built one for standalone and it worked perfectly on the larger db, still tweaking on the plugin importer.

Version 0, edited 3 years ago by anointed (next)

comment:4 johnjamesjacoby3 years ago

  • Milestone changed from Awaiting Review to 2.1
  • Version 2.0 deleted

There is a redux of the migration tool in the works, along with migrators from several other popular forum packages.

Bumping to 2.1 so we can reevaluate this after 2.0.

comment:5 Sadr2 years ago

I can provide database files for a 20k BuddyPress legacy (1.x) forum.

comment:6 cnorris232 years ago

  • Keywords dev-feedback added; reporter-feedback removed

I know from jjj's previous comment that this is getting some work, but just in case, I had a thought. Change the foreach statements from

foreach( $topics as $topic )

to

foreach( $topics as $key => $topic )

and at the end of the foreach run

unset( $topics[$key] )

then for good measure, run

unset( $topics )

somewhere soon after the foreach.

This should help reduce memory usage as we chug along through the import.

@jjj, if you have no objections, and it's help is still needed on this, I'd be glad to work up a patch for testing.

comment:7 johnjamesjacoby2 years ago

  • Resolution set to fixed
  • Status changed from new to closed

First pass: replace bbPress 1.1 importer with bbConverter by Adam Ellis.

  • Handles large bbPress installations much better than original importer
  • Better potential for new converters in future
  • Breathes life into abandoned bbConverter project
  • See #1592
  • Props Adam Ellis for bbConverter
  • @todo: Settings/UI/code clean-up, shake vigorously, lather, rinse, repeat
Note: See TracTickets for help on using tickets.