Skip to:
Content

Opened 3 years ago

Closed 2 years ago

#1592 closed defect (fixed)

bbPress Standalone Importer breaks if source forum is too big

Reported by: ali3n0 Owned by:
Milestone: 2.1 Priority: normal
Severity: normal Version:
Component: Importers Keywords: dev-feedback
Cc:

Description

Seems there's a memory leak somewhere, given that even growing allowed memory indefinitely it still breaks pretty much at the same point. You can see the error in the attachment

Attachments (1)

bbpress20_migrate_fatal_error.jpg (230.7 KB) - added by ali3n0 3 years ago.

Download all attachments as: .zip

Change History (8)

comment:2 cnorris233 years ago

  • Keywords reporter-feedback added

Is anyone with this problem willing/able to provide some database files to test this out? It would take some work to produce forum data of that magnitude manually. Alternatively, does anyone know of a plugin that one could use to create database info, similar to http://wordpress.org/extend/plugins/demo-data-creator/?

comment:3 anointed3 years ago

@cnorris23

I don't have a bbpress forum, but can provide a vbulletin forum db.
Threads: 60,982, Posts: 295,135, Members: 26,800

We also have access to forums with 10million+ posts and 500k members if that would help.

*I've currently got one of the vb devs working on building me a vb importer for bbpress plugin. He built one for standalone and it worked perfectly on the larger db, still tweaking on the plugin importer.

Last edited 3 years ago by anointed (previous) (diff)

comment:4 johnjamesjacoby3 years ago

  • Milestone changed from Awaiting Review to 2.1
  • Version 2.0 deleted

There is a redux of the migration tool in the works, along with migrators from several other popular forum packages.

Bumping to 2.1 so we can reevaluate this after 2.0.

comment:5 Sadr2 years ago

I can provide database files for a 20k BuddyPress legacy (1.x) forum.

comment:6 cnorris232 years ago

  • Keywords dev-feedback added; reporter-feedback removed

I know from jjj's previous comment that this is getting some work, but just in case, I had a thought. Change the foreach statements from

foreach( $topics as $topic )

to

foreach( $topics as $key => $topic )

and at the end of the foreach run

unset( $topics[$key] )

then for good measure, run

unset( $topics )

somewhere soon after the foreach.

This should help reduce memory usage as we chug along through the import.

@jjj, if you have no objections, and it's help is still needed on this, I'd be glad to work up a patch for testing.

comment:7 johnjamesjacoby2 years ago

  • Resolution set to fixed
  • Status changed from new to closed

First pass: replace bbPress 1.1 importer with bbConverter by Adam Ellis.

  • Handles large bbPress installations much better than original importer
  • Better potential for new converters in future
  • Breathes life into abandoned bbConverter project
  • See #1592
  • Props Adam Ellis for bbConverter
  • @todo: Settings/UI/code clean-up, shake vigorously, lather, rinse, repeat
Note: See TracTickets for help on using tickets.